Research question
The central question driving this analysis is why a model with a high win rate—or more precisely, lower Brier score indicating better probability estimation—can still produce a negative return on investment. Derived from the profitability_paradox candidate type, this question is fundamental to model evaluation: it separates pure prediction accuracy from betting profitability. Understanding the divergence between calibration and ROI is critical for users who rely on AI models for decision-making, as a well-calibrated model may still suffer from overconfident low-probability bets or adverse selection in odds markets.
Evaluation period
The evaluation window covers fixtures from 2026-07-01 through 2026-07-31, a 31-day period. All picks in this window are settled and were generated from pre-kickoff model snapshots, ensuring no hindsight bias. The dataset does not restrict to specific leagues; as indicated by the empty pilot_leagues_label, it includes matches across multiple competitions. However, the absence of league-level granularity limits the ability to assess model performance in homogeneous market conditions.
Sample size
The research draws on 419 settled picks, classified as "Developing evidence" by Alpha Bench's minimum sample framework. While this sample supports preliminary trend identification, it falls well short of the thousands of picks typically needed for robust ROI inferences. Per-model samples are even more constrained: grok-build-0.1 has 52 picks, gpt-5-mini has 28, deepseek-v4-pro has 230, and deepseek-v4-flash has 109. The gpt-5-mini subsample of 28 picks is below the minimum sample threshold of 30, as flagged by vector-dataset-status. A larger multi-month sample would be necessary to confirm whether observed ROI differentials persist.
Model comparison at a glance
| Model | Settled picks | Brier | Log Loss | ROI (%) | |-------|--------------|-------|----------|---------| | grok-build-0.1 | 52 | 0.2425 | 0.674 | +3.2 | | gpt-5-mini | 28 | 0.2489 | 0.6887 | +1.1 | | deepseek-v4-pro | 230 | 0.3397 | 1.1811 | -10.6 | | deepseek-v4-flash | 109 | 0.3551 | 1.0853 | -14.4 |
The table reveals a striking inversion: the model with the best Brier and log loss (grok-build-0.1) achieves the only positive ROI, while gpt-5-mini, with a nearly comparable Brier, barely crosses into positive territory. In contrast, the two deepseek models display substantially worse Brier scores and deeply negative returns. This spread—from +3.2% to -14.4% ROI—underscores that calibration alone cannot predict profitability, especially when sample sizes vary widely.
Calibration and probability accuracy
Calibration analysis from vector-calibration exposes systematic overconfidence in moderate-to-high probability ranges. For example, in the 60–70% predicted probability decile, the average predicted probability was 74 while the actual outcome rate was only 53. Similarly, the 70–80% decile shows a predicted mean of 96 against an actual of 25. This pattern of overestimation in the 60–80% bucket can erode expected value even when a model's overall Brier appears decent. The top decile (90–100%) shows predicted 34 but actual 23, indicating modest overconfidence at the extreme. Such miscalibration directly impacts ROI because wagers placed on overvalued probabilities will suffer negative expected value over time, regardless of the model's win rate.
Risk, drawdown, and bankroll behavior
Examining risk metrics reveals substantial drawdown variance across models, further divorcing Brier from realized equity outcomes. The top-ranked model, grok-build-0.1 (index 1 in bankroll_drawdown), encountered a maximum drawdown of 373.93 currency units from a peak of 1017.07, closing at 871.24. That drawdown represents a peak-to-trough decline of +36.8%, a non-trivial risk even though the model ended with positive ROI. In contrast, deepseek-v4-pro (index 0) experienced a drawdown of 605.18 from a peak of 1094.5, leaving current equity at just 489.32—an +55.3% peak-to-trough loss. This asymmetry illustrates that even a "profitable" model in this small sample carries high volatility; historical returns are not indicative of future stability.
Model-by-model notes
grok-build-0.1: With a Brier of 0.2425 and ROI of +3.2%, this model leads the benchmark by combining the best calibration score with the only positive return. However, its sample of only 52 picks and a max drawdown of 373.93 caution against overinterpreting its rank. The consistency score of 32.3 suggests moderate day-to-day stability.
gpt-5-mini: This model posts a close second in Brier (0.2489) and log loss (0.6887), with a slim positive ROI of +1.1%. However, its sample size of only 28 picks is the smallest in the set, below the minimum threshold for reliable inferences. Its max drawdown of 1010.88 and current equity of 114.89 appear less severe, but the low pick count renders any performance metric provisional.
deepseek-v4-pro: As the most sampled model with 230 picks, it provides the most statistically stable estimates—yet its Brier of 0.3397 and ROI of -10.6% are the worst among the four aside from flash. Its calibration error is evident in the calibration deciles, and its max drawdown of 605.18 (nearly wiping out starting equity) indicates extreme risk.
deepseek-v4-flash: With a Brier of 0.3551 and ROI of -14.4%, this model ranks lowest overall. It has a moderate sample of 109 picks and a max drawdown of 560. The negative return, combined with a peak equity of 1318.45 and current equity of 1302.6, suggests persistent underperformance.
What this means for readers
The benchmark demonstrates that Brier and ROI diverge meaningfully across models, even within a modest sample. No model guarantees future profitability, and the best-calibrated model may still incur large drawdowns. Readers should consult individual model profiles (such as the grok-build-0.1 model profile and deepseek-v4-pro model profile) for detailed performance breakdowns, and review the methodology to understand how Alpha Bench freezes snapshots and aggregates metrics. Earlier research on calibration for Both Teams to Score and match result calibration provides additional context. No betting advice is offered.
Methodology
Alpha Bench freezes pre-kickoff model snapshots at a fixed cutoff before each fixture, ensuring no look-ahead bias. Each model's predictions are compared against settled outcomes to compute Brier score and log loss. ROI is calculated on a standardized starting bankroll with flat stakes. Maximum drawdown, peak equity, and current equity are tracked from daily model portfolio snapshots. All metrics are computed according to the Alpha Bench methodology version 2.1. The analysis aggregates picks across multiple leagues (pilot leagues are not explicitly segmented in this dataset) and applies a minimum sample threshold of 30 picks before reporting performance; the gpt-5-mini model's 28 picks falls below this threshold.
Limitations
This analysis is constrained by several factors. First, the total sample of 419 settled picks is modest, and per-model samples range from 28 to 230, with the smallest subsample (gpt-5-mini) yielding unreliable ROI estimates. Second, no closing-line value (CLV) data is available for any model, preventing analysis of a critical dimension of profitability. Third, the evaluation period covers only one month (July 2026), which may capture temporary market inefficiencies or model performance fluctuations. Fourth, the absence of league-specific breakdowns limits the ability to assess model specialization. Finally, model version changes occurred for all four models during the period, as recorded in vector-version-changes, meaning the benchmark compares evolving algorithms rather than static versions. These limitations do not invalidate the findings but argue against overgeneralization.
Reproducible data references
The data underlying this analysis comes from Alpha Bench's frozen dataset, captured at 2026-07-24T16:00:02Z with article run ID 2026-07-24-vector-001. Key sources include: vector-dataset-status (sample size labels and insufficiency flags), vector-model-comparison (model rankings and evaluation period), vector-risk-metrics (drawdown and equity data), vector-calibration (calibration deciles), and vector-version-changes (model update logs). Internal metrics from historical_metrics (e.g., gpt-5-mini's Brier and ROI) are directly traceable to the package's model records. Readers can verify claims by visiting the model profile pages (e.g., /models/deepseek-v4-flash) and the methodology page (/methodology), where full definitions and computational methods are disclosed. Transparency blocks for individual fixtures (not included in this article) would allow per-pick verification of predictions and outcomes.