Research question
How do grok-build-0.1 and gpt-5-mini compare in terms of prediction accuracy, calibration, and profitability in a head-to-head benchmark? This question, grounded in the primary_keyword and search_intent, matters for model evaluation because early cross-model comparisons can mislead when sample sizes are small. The candidate_type is model_comparison, and the research focuses on a direct model comparison rather than a specific league or market.
The analysis explores whether the two models exhibit meaningful differences in Brier score, log loss, and return on investment, and whether any observed gaps are reliable given the data constraints. A head-to-head benchmark is valuable for users deciding between models, but statistical validity depends on sufficient settled picks and consistent evaluation conditions.
Evaluation period
The frozen evaluation window spans 2026-07-01 through 2026-07-31. The same period is reflected in the long-term patterns benchmark period (2026-07-01 through 2026-07-31). No pilot leagues were defined (), so the data is not restricted to any single competition.
Only settled picks with pre-kickoff snapshots are included. Any picks beyond the data cutoff at 2026-07-07T21:59:27Z are excluded. The evaluation window is brief (one month) and the number of settled picks is extremely low, which limits the period’s ability to capture consistent model behavior.
Sample size
The sample size supporting this analysis is 4, which is labeled as “Insufficient sample”. This figure falls far below the typical minimum of 30 picks required for statistically sound inference. The vector-model-comparison source also reports a sample of 4.
With only four settled picks across all models, any observed differences in accuracy or profitability could easily be driven by noise rather than true model skill. Confidence in conclusions would be strengthened by a sample at least an order of magnitude larger, ideally spanning multiple leagues and market types.
Model comparison at a glance
| Model | Settled picks | Brier | Log loss | ROI (%) | |-------|---------------|-------|----------|---------| | grok-build-0.1 | 11 | 0.1614 | 0.5015 | -32.25 | | gpt-5-mini | 4 | 0.1902 | 0.5688 | 47.65 | | deepseek-v4-flash | 17 | 0.3052 | 0.8158 | -41.44 | | deepseek-v4-pro | 54 | 0.3384 | 1.1526 | -23.79 |
The spread between best Brier (grok-build-0.1) and worst Brier (deepseek-v4-pro) is substantial, but the ROI pattern is inverted: the model with the lowest Brier has a negative ROI, while gpt-5-mini, with a higher Brier, yields the only positive ROI. This paradox underscores that Brier alone does not determine profitability, and with such tiny sample sizes, these rankings are fragile.
Calibration and probability accuracy
The calibration deciles from vector-calibration provide a window into systematic miscalibration across all models. For example, in the highest decile (30-40%), the predicted probability is 35 but the actual outcome rate is 10, indicating extreme overconfidence. Similarly, the 80–90% decile shows a predicted of 97 versus actual 0.
These patterns suggest that the models (as a group) tend to overestimate probabilities at the high end, which can inflate expected value calculations and mislead bettors relying on confidence thresholds. However, the calibration data is aggregated from 69 fixtures and not broken down by model, so we cannot attribute miscalibration specifically to grok-build-0.1 or gpt-5-mini. The small model-level sample further dilutes calibration accuracy.
Risk, drawdown, and bankroll behavior
Risk metrics from vector-risk-metrics reveal contrasting drawdown profiles. For grok-build-0.1, the maximum drawdown (peak-to-trough) is 373.93 units, with a peak equity of 1017.07 and a current equity of 893.34. Its average daily P&L is -9.46.
In contrast, gpt-5-mini shows a lower maximum drawdown of 966.55, with peak equity 1123.48 and current equity 173.23. While gpt-5-mini has a smaller drawdown, its average daily P&L is -14.01, more negative than grok-build-0.1’s. This illustrates that low drawdown does not guarantee steady positive returns. deepseek-v4-pro experienced the severest drawdown (510.36), reminding that high-risk profiles can erode bankrolls quickly.
Model-by-model notes
grok-build-0.1 shows the best Brier (0.1614) and log loss (0.5015) among all models, yet its ROI is negative (-32.3). Its settled picks count (11) is small, so the accuracy metrics may be optimistic. The drawdown of 373.93 indicates substantial risk despite good calibration signals. This model’s performance should be viewed as preliminary.
gpt-5-mini has a higher Brier (0.1902) and log loss (0.5688), but an exceptionally high positive ROI (+47.7). With only 4 settled picks, this ROI is unreliable. Its drawdown is lower (966.55), but the average daily loss is larger. The inconsistency between accuracy and profit highlights the danger of small-sample evaluations.
deepseek-v4-flash (not the focus of this head-to-head) has the third-best Brier (0.3052) and the worst ROI (-41.44). Its settled picks (17) are the second highest, but the negative return persists.
deepseek-v4-pro has the highest Brier (0.3384) and the largest sample (54), yet its ROI is negative (-23.79). The drawdown (510.36) is the deepest, indicating that even with more data, performance is poor.
What this means for readers
The benchmark shows that grok-build-0.1 and gpt-5-mini exhibit contrasting accuracy-profitability profiles, but the sample sizes are far too small to draw robust conclusions. The calibration data indicates systematic overconfidence across all models, which should be considered when interpreting predicted probabilities. Readers are encouraged to verify these claims by visiting the model profiles for grok-build-0.1 and gpt-5-mini, and by reviewing the full methodology and metric definitions. No betting advice or guarantees should be inferred from this preliminary data.
Methodology
Alpha Bench freezes pre-kickoff snapshots of model predictions, then evaluates them against settled outcomes using Brier score, log loss, and ROI. Drawdown and bankroll trajectories are computed from daily equity changes. Samples are aggregated across all available competitions, but no league-specific breakdowns exist in this dataset. The methodology version is 2.1. Full details are available on the methodology page.
Limitations
This analysis is constrained by an extremely small sample of 4 settled picks, labeled “Insufficient sample”. The pilot leagues list is empty, meaning no league-specific context is available. Market coverage is limited (CLV data is null for all models), and calibration deciles are not model-specific. Version changes are present for all models (see vector-version-changes), which may have affected consistency during the evaluation period. The data does not prove which model is superior; chance alone could explain the observed differences. These limitations prevent generalization to out-of-sample periods or other markets.
Reproducible data references
This article uses the following Alpha Bench data sources: 2026-07-07-vector-010, data cutoff 2026-07-07T21:59:27Z, methodology version 2.1. Verified context entries include vector-dataset-status, vector-model-comparison, vector-calibration, vector-risk-metrics, and vector-version-changes. Readers can reproduce the analysis by accessing the frozen snapshot at the competition hub and verifying individual predictions on fixture transparency blocks. Model profiles linked from internal_links provide additional per-model metrics.