Research question
Why does a model achieving a lower Brier score — often interpreted as "higher win rate" — not always produce a positive return on investment (ROI) in AI football betting? This analysis examines the profitability paradox using the Alpha Bench pilot-league dataset from 2026-06-01 to 2026-06-30, focusing on the relationship between prediction accuracy, bankroll drawdown, and net returns.
Evaluation period
The frozen evaluation window spans from 2026-06-01 through 2026-06-30.
Sample size
The analysis is supported by 384 settled picks across four AI models. This is classified as an Early sample. The minimum sample threshold of 30 picks is met, but individual model sample sizes range from 18 to 241 picks, introducing variance.
Findings
The data reveals a clear disconnect between predictive accuracy and profitability. The model with the lowest Brier score (0.2311) — grok-build-0.1 — posted a negative ROI of -35.3%. In contrast, deepseek-v4-pro, with a higher Brier of 0.3201, showed a less negative ROI of -13.4%. deepseek-v4-flash and gpt-5-mini followed similar patterns. Notably, all models experienced peak equity in positive territory, but subsequent drawdowns eroded gains. deepseek-v4-pro suffered a max drawdown of {{context.vector-risk-metrics.bankroll_drawdown[2].max_drawdown}} (+80.8% of peak). The calibration analysis shows systematic overconfidence: in the highest predicted probability decile (90-100%), actual outcomes occurred only {{context.vector-calibration.calibration_deciles[9].actual}}% of the time against a predicted probability of {{context.vector-calibration.calibration_deciles[9].predicted}}%. This overconfidence likely leads to poor bet sizing and increased drawdown. No closing-line value (CLV) data was available for this period, limiting analysis of market timing.
Methodology
Alpha Bench freezes model predictions before kickoff, simulating fixed stake sizes. Metrics include Brier score, log loss, ROI, drawdown, and calibration deciles. All data is from pre-kickoff snapshots to avoid post-hoc bias. Methodology version 2.1. See full methodology for details.
Limitations
The sample size of 384 picks is early (Early sample). Two models (grok-build-0.1 and gpt-5-mini) have fewer than 20 settled picks, making their metrics statistically unstable. CLV data was unavailable, so we could not assess whether models were consistently beating the market. The pilot covers five leagues (Bundesliga, Champions League, LaLiga, Premier League, Serie A) and may not generalize. No market-type breakdown was available. These results should not be interpreted as final evidence against the profitability of AI models.
Reproducible data references
Data sources: Alpha Bench dataset (article_run_id 2026-06-23-vector-005). Key verified context entries: vector-dataset-status, vector-model-comparison, vector-risk-metrics, vector-calibration. Internal metrics: model_daily_metrics for each model (e.g., grok-build-0-1:roi_percent). Readers can verify claims on individual model profiles (grok-build-0.1, deepseek-v4-pro, deepseek-v4-flash, gpt-5-mini). Metric definitions are documented at /trust. Full methodology at /methodology. Data source overview at /data-sources.