Research question
The analytical question is how gpt‑5‑mini and grok‑build‑0.1 compare in football prediction accuracy and profitability during Alpha Bench’s early pilot phase. Given that both models are newer entrants (the data package reports version changes for both), understanding their relative calibration, return on investment, and risk profiles matters for model evaluation even before large samples accumulate. This analysis uses the model_comparison candidate type from model_comparison to examine the head‑to‑head vs. the broader field.
Evaluation period
The frozen evaluation window runs from 2026-07-01 through 2026-07-31. This window covers July 2026. Only settled picks with pre‑kickoff snapshots are included; no live or post‑match adjustments. The pilot leagues are not specified in this data package (the field pilot_leagues_label is empty), so the results are league‑agnostic.
Sample size
The sample supporting this analysis consists of 34 settled picks (label: Early sample). While the system’s minimum sample size of 30 is exceeded, this is still an early sample. The findings would be strengthened by several hundred additional picks. All INTERNAL_STATISTIC claims are traced to vector‑dataset‑status and model‑specific settled_picks counts.
Model comparison at a glance
| Model | Settled Picks | Brier Score | Log Loss | ROI (%) | |-------|---------------|-------------|----------|---------| | gpt‑5‑mini | 34 | 0.2322 | 0.6524 | -2.11 | | grok‑build‑0.1 | 59 | 0.2342 | 0.6559 | -0.43 | | deepseek‑v4‑pro | 230 | 0.3397 | 1.1811 | -10.63 | | deepseek‑v4‑flash | 122 | 0.3619 | 1.0944 | -16.67 |
The spread between the best Brier (gpt‑5‑mini, 0.2322) and worst (deepseek‑v4‑flash, 0.3619) is notable, but ROI tells a different story: grok‑build‑0.1’s ROI of -0.43% is the least negative, despite a marginally worse Brier. This highlights that raw probability accuracy does not directly translate to betting profitability due to market odds and overround.
Calibration and probability accuracy
The calibration deciles from the full settled fixture set (353 fixtures) reveal systematic overconfidence at high predicted probabilities. For example, events predicted in the 90‑100% decile (45 observations) had an actual outcome rate of 27 against a mean predicted probability of 44. This gap suggests models are generally too aggressive in high‑confidence bins, which is a known pattern that can erode expected value.
The lower deciles (e.g., 20‑30% predicted: actual 25, predicted 16) show less severe miscalibration but still directional overconfidence. Because calibration data is aggregated across models and leagues, these signals should be interpreted cautiously when applied to individual model performance.
Risk, drawdown, and bankroll behavior
Looking at the bankroll drawdown data for the full model set, grok‑build‑0.1 shows a maximum drawdown of 373.93 units (peak equity 1017.07), while gpt‑5‑mini’s drawdown is lower at 560 (peak 1318.45). However, grok‑build‑0.1’s current equity of 859.19 exceeds gpt‑5‑mini’s 1271.45, reflecting its higher starting bankroll or fewer settled picks. The deepseek‑v4‑pro suffered the worst drawdown of 656.86 (+60% of peak), indicating substantial equity risk even with a larger sample.
Model-by-model notes
gpt‑5‑mini (34 settled picks) achieves the best Brier score (0.2322) and log loss (0.6524) among the four models, but its ROI of -2.1% is the second worst, dragged down by small sample volatility. The calibration sample for this model is not separately reported, so we cannot isolate its calibration error from the aggregate.
grok‑build‑0.1 (59 settled picks) shows the least negative ROI (-0.4%) while having a slightly worse Brier (0.2342) and log loss (0.6559) than gpt‑5‑mini. Its consistency score from historical metrics is 33.3, marginally higher than gpt‑5‑mini’s 42, suggesting more stable prediction patterns per fixture.
What this means for readers
This early benchmark does not prove superiority of either model. The small sample size (34 picks for the primary comparison) means that chance variation can heavily influence both Brier and ROI. Readers should consult the model profiles for gpt‑5‑mini and grok‑build‑0.1 to see live updated metrics. Additional context on metric definitions is available on the trust page. No betting advice is offered; these are observational benchmarks.
Methodology
Alpha Bench methodology version 2.1 governs this analysis. Pre‑kickoff snapshots are frozen to avoid look‑ahead bias. Brier and log loss are computed from predicted vs. actual outcomes on a per‑fixture basis, with ROI calculated as (P&L / stake) × 100. Drawdown is measured from peak equity. The pilot sample aggregates all settled picks across available leagues without filtering by competition. Full methodology is described here.
Limitations
First, the sample size of 34 is an early sample; the data package itself labels it as "Early sample". Second, closing line value (CLV) is not recorded for any model in this period (all CLV fields are null), which is a critical missing piece for assessing whether model predictions have edge against efficient markets. Third, the pilot leagues are unspecified, so results may not generalize to a specific league or market type. Finally, the calibration deciles are aggregate across models and cannot be attributed to any single model in the comparison.
Reproducible data references
This analysis uses data from article run 2026-07-27-vector-001, with a cutoff at 2026-07-27T16:00:02Z. Verified context sources include: vector-dataset-status (sample size), vector-model-comparison (benchmark leaderboard), vector-calibration (calibration deciles), vector-risk-metrics (drawdown and equity), and vector-long-term-patterns (benchmark period). Per‑model historical metrics are sourced from the historical_metrics array in the data package (e.g., roi_percent, brier). Readers can verify claims by reviewing the source data on the data sources page and individual model profiles linked above.