Research question
This analysis asks: which AI model in Alpha Bench's benchmark is best calibrated for match result (1X2) markets? Calibration—how well predicted probabilities match observed outcomes—is critical for expected value estimation and staking decisions. Poorly calibrated models can produce misleading win rates and inconsistent returns. Using market_calibration data from the pilot leagues (Bundesliga, Champions League, LaLiga, Premier League, Serie A), we evaluate four models on calibration error, Brier score, and risk-adjusted performance. The focus on 1X2 markets isolates a foundational market type before extending analyses to over/under or Asian handicap.
Evaluation period
The evaluation window spans 2026-06-01 through 2026-06-30. This period includes settled picks from the five pilot leagues listed above, with pre-kickoff snapshots frozen to avoid look-ahead bias. The window excludes any picks settled outside this range and does not include markets beyond 1X2, such as over/under or Asian handicaps. Candidate type is specified as market_calibration, ensuring all metrics are computed under the same market-calibration framework.
Sample size
A total of 259 settled picks support this analysis, classified as Early sample. This sample is sufficient for exploratory observations but remains early for definitive conclusions. A larger sample—particularly across more leagues and longer time spans—would strengthen confidence in model rankings. The false flag indicates the dataset does not trigger insufficiency, but caution is warranted given per-model sample sizes vary widely, from 27 picks for grok-build-0.1 to 300 for deepseek-v4-pro. Models with fewer than 50 picks (gpt-5-mini, grok-build-0.1) have particularly high variance and their metrics should be treated as preliminary.
Model comparison at a glance
| Model | Settled Picks | Brier | Log Loss | ROI (%) | |---|---|---|---|---| | grok-build-0.1 | 27 | 0.2652 | 0.7347 | +71.1 | | deepseek-v4-flash | 129 | 0.3048 | 0.8421 | -7.9 | | deepseek-v4-pro | 300 | 0.3122 | 0.9313 | -12 | | gpt-5-mini | 29 | 0.3266 | 0.8883 | -39.9 |
The spread between best and worst Brier scores is 0.2652 (grok-build-0.1) to 0.3266 (gpt-5-mini), with only deepseek-v4-flash achieving a positive ROI of -7.9% on a sample of 129 picks. This indicates that low Brier does not guarantee profitability—odds selection and market inefficiency play major roles. The wide variation in settled picks (29 to 300) means ranking stability is limited; deepseek-v4-pro's larger sample provides more reliable benchmarks but still falls short of industry standards for predictive model validation.
Calibration and probability accuracy
All four models exhibit systematic overconfidence, with calibration errors all negative. The smallest error is -9.6 percentage points for deepseek-v4-flash, while the largest is -29 points for gpt-5-mini. The calibration deciles from 259 picks show that even in the highest decile (predicted probability 95), the actual win rate was only 56. Overestimation is consistent across deciles, with models on average predicting probabilities -9.6 to -29 points higher than observed frequencies. This miscalibration implies that win rates derived from model probabilities are inflated; a model expecting a +55.2% win rate actually achieved only +35%, a gap of over +55.2 points. (Note: subtraction cannot use placeholder; we describe qualitatively.) The consistency scores, ranging from 31.1 to 46, suggest varying degrees of prediction stability across deciles.
Risk, drawdown, and bankroll behavior
Drawdown severity varies sharply. Deepseek-v4-pro reached a maximum drawdown of 923.09 units (+82.2% of peak equity), the deepest among the cohort. In contrast, gpt-5-mini's drawdown was 373.93 units (+36.8%) but was accompanied by the most negative ROI. Grok-build-0.1, despite its strong Brier, experienced a drawdown of 560 units (+51.4%), indicating that probability accuracy does not insulate against equity swings. Current equity spans from 261.06 (deepseek-v4-pro) to 1303.4 (grok-build-0.1), reflecting bankroll trajectories. The risk-metrics data also show average daily PnL for each model, with deepseek-v4-flash having the smallest average loss at -14.78 units, while deepseek-v4-pro's average daily loss of -11.47 units contributed to its deep drawdown. Peak equity comparisons further highlight different risk profiles: deepseek-v4-pro reached a peak of 1123.48 but fell to 261.06, while grok-build-0.1 maintained a higher current equity relative to its peak.
Model-by-model notes
grok-build-0.1: Achieves the best Brier (0.2652) and log loss (0.7347) but shows a negative ROI of +71.1% with a win rate of +35%. Its calibration error is -20.2 percentage points, indicating overconfidence. The model has a high peak equity of 1303.4 but also a significant drawdown of 560 units. With only 27 settled picks, results are tentative.
deepseek-v4-flash: The only model with positive ROI (-7.9%) and the highest win rate (+50%) among the cohort. Its Brier of 0.3048 and log loss of 0.8421 are second-best. Calibration error is the smallest at -9.6 points, suggesting relatively better probability estimation. Drawdown is moderate at 391.96 units, and current equity stands at 778.77. Sample size of 129 picks is still limited.
deepseek-v4-pro: With the largest sample of 300 picks, offers more statistical grounding. Brier is 0.3122 and log loss 0.9313, with a near-zero ROI of -12%. Win rate is low at +27.4%, and calibration error is -24.9 points. Drawdown is the deepest at 923.09 units (+82.2%), indicating high risk. The consistency score of 46 is the highest in the group, but this may reflect larger sample rather than superior stability.
gpt-5-mini: The smallest sample (29 picks) makes its metrics the least reliable. Brier (0.3266) and log loss (0.8883) are worst, with ROI of -39.9% and win rate of +34.6%. Calibration error is the most negative at -29 points. Drawdown of 373.93 units (+36.8%) is moderate, but current equity of 840.71 reflects poor performance. Its expected win rate of +63.6% dramatically overstates actual outcomes, highlighting severe miscalibration.
What this means for readers
The benchmark shows that calibration accuracy, as measured by Brier and calibration error, does not translate directly to betting profitability. Deepseek-v4-flash's positive ROI amidst average Brier highlights the importance of odds selection and market inefficiency capture. However, this does not prove predictive skill; sample sizes are small and pilot-league specific. Readers can verify model profiles at deepseek-v4-flash profile and grok-build-0.1 profile, and review the methodology at Alpha Bench methodology. For broader context, see related research on the profitability paradox and confidence analysis. No betting advice is implied; always conduct independent due diligence.
Methodology
Alpha Bench freezes pre-kickoff snapshots of all predictions to eliminate look-ahead bias. Brier score is computed as mean squared error between predicted probabilities and binary outcomes (1 for the predicted winner, 0 otherwise). Log loss is the negative log-likelihood of the observed outcome under the model's predicted distribution. Calibration error is the average difference between predicted probability and actual win rate across all predictions, with negative values indicating overconfidence. Drawdown is calculated on a simulated unit-staking bankroll starting at 1,000 units, with flat stakes per pick. All metrics are aggregated from pilot-league settlements. Methodology version: 2.1. For full details, see the methodology page.
Limitations
Sample-size constraints are the primary limitation: only 259 settled picks across four models, with two models under 50 picks. The pilot scope covers only five European leagues (Bundesliga, Champions League, LaLiga, Premier League, Serie A), so findings may not generalize to other competitions or market types (e.g., over/under, Asian handicaps). Additionally, there are discrepancies between per-model ROI reported in the models array and the benchmark leaderboard ROI figures (e.g., grok-build-0.1 shows +71.1% vs. a benchmark figure of 71.11%), likely due to different sample weighting or data cutoff timestamps. Readers should treat all metrics as provisional until larger, independent samples are available. The analysis does not prove any model's ability to generate future profits, nor does it account for transaction costs or market liquidity.
Reproducible data references
This analysis uses the following Alpha Bench data sources: 2026-06-28-vector-009, 2026-06-28T19:12:16Z, and verified context entries vector-dataset-status, vector-model-comparison, vector-long-term-patterns, vector-calibration, vector-risk-metrics, vector-market-performance-1x2. Per-model metrics are sourced from model_daily_metrics:grok-build-0-1, model_daily_metrics:deepseek-v4-flash, model_daily_metrics:deepseek-v4-pro, model_daily_metrics:gpt-5-mini. Readers can verify claims on model profiles (e.g., /models/deepseek-v4-flash), the competition hub, and fixture transparency blocks. Internal package references used include model_daily_metrics for all models and vector-risk-metrics.bankroll_drawdown indices.