The research answer is that ROI, probability accuracy and closing-line value do not align consistently in this early sample: deepseek-v4.1-flash has the highest reported ROI, while ling-3.1-flash has the lowest Brier score and the most negative ROI. That gap matters because a favorable result on settled picks is not, by itself, evidence of better-calibrated probabilities or repeatable profitability. The frozen package includes 51 settled picks and labels the sample Early sample.
Research question
The question is whether AI football models with stronger short-sample results also show stronger evidence of probability accuracy or market value. It is grounded in the package’s candidate type, profitability_paradox, and the stated search intent: explaining the relationship between win rate, ROI and closing-line value. The focus is a cross-model profitability comparison, not a league- or market-specific ranking.
This matters because these measures answer different questions. ROI records the return in the observed sample, Brier score summarizes probability error, and CLV records the relationship between the selected price and the closing price. The package does not provide a model-level win-rate field, so this analysis cannot test whether a higher win rate corresponds to higher profitability; it can show why ROI alone is not enough to establish model quality.
Evaluation period
The frozen evaluation window runs from 2026-10-01 through 2026-10-31. The comparison covers the models and settled picks represented in the supplied benchmark data, with the overall sample labeled Early sample. The package lists no pilot-league names, so the scope cannot be narrowed to a named competition.
The period end is later than the package’s data cutoff, 2026-10-18T16:00:02Z. That makes the reported results a frozen, partial view of the stated window rather than a completed evaluation through its scheduled end. The supplied predictions and odds-snapshot arrays are empty, so this article cannot independently reconstruct individual pre-kickoff forecasts or prices from those arrays.
Sample size
The analysis rests on 51 settled picks, labeled Early sample; the dataset-status entry says insufficient data is false under its stated threshold. That status permits publication under the package’s rules, but the label still calls for caution. Per-model counts vary substantially, from 2 for ling-3.1-flash to 16 for deepseek-v4.1-flash.
This is enough to describe the reported spread, not to establish stable rankings. Several models have only a handful of settled picks, and even the largest model-specific sample remains limited for general conclusions. Confidence would strengthen with more settled observations across the same evaluation design and with a complete window; the current data cannot show whether the ordering persists.
Model comparison at a glance
The table includes every model in the package. Brier and ROI show different kinds of performance, while CLV adds a market-price comparison; none of these columns is a win-rate measure.
| Model | Settled picks | Brier | ROI | CLV | |---|---:|---:|---:|---:| | ling-3.1-flash | 2 | 0.0712 | -100 | +0 | | solar-mini4 | 4 | 0.2073 | +34 | -1.9 | | qwen3.8-omni-flash | 4 | 0.2073 | +34 | -1.9 | | deepseek-v4.1-flash | 16 | 0.2372 | +54 | +0 | | apodex-1.1-mini | 10 | 0.2379 | -24.1 | +0.3 | | gpt-6-luna-pro | 5 | 0.268 | -26.5 | +3.8 | | nemotron-3.5-lightning | 5 | 0.2695 | -33 | +3.4 | | claude-haiku-5.5 | 5 | 0.2695 | -33 | +3.4 |
The most striking contrast is that ling-3.1-flash records the lowest Brier score, 0.0712, alongside ROI of -100; its settled-pick count is just 2. Deepseek-v4.1-flash reports ROI of +54 on 16 picks, with a higher Brier score of 0.2372. These results illustrate disagreement among metrics, not proof that either model will retain its observed position.
Calibration and probability accuracy
The aggregate calibration table compares predicted probabilities with observed outcomes across probability bands. For example, the band labeled 70-80% has an average predicted probability of 73 and an observed rate of 38. In the band labeled 80-90%, the predicted value is 82, compared with an observed rate of 25.
Those gaps suggest that aggregate probabilities did not track observed outcomes evenly across the reported bands. They do not identify which model is over- or under-estimating, because the supplied calibration entries are not broken out by model, league or market. A win rate would also be a different quantity: without model-level win rates and the underlying odds context, it cannot establish expected value or explain why ROI differs. The market-performance entry contains no market rows, so market-specific calibration conclusions are unavailable.
Risk, drawdown, and bankroll behavior
The bankroll rows add a risk dimension to the ROI comparison. Deepseek-v4.1-flash has reported current equity of 1006.8 and maximum drawdown of 5, while apodex-1.1-mini has current equity of 984.11 and maximum drawdown of 28.81. The second model’s ROI is negative in the comparison, and its drawdown is larger than the values reported for several other models.
These figures describe the observed bankroll paths, not the risk of future results. Solar-mini4 and qwen3.8-omni-flash each have reported maximum drawdown of 5 and current equity of 990, alongside matching summary results in the benchmark. Their short records do not establish a steadier long-run profile. The data supports comparing the reported paths, but not treating lower observed drawdown as a guarantee of lower future risk.
Model-by-model notes
Ling-3.1-flash combines a Brier score of 0.0712 with ROI of -100 and CLV of +0. Its sample is 2 picks, so the sharp contrast between its probability score and return should be read as a small-sample result, not a durable ranking. The package’s version-change record also reports changes for this model, which further cautions against treating the observed results as a timeless model description.
Solar-mini4 has Brier 0.2073, ROI +34 and CLV -1.9 across 4 settled picks. Qwen3.8-omni-flash reports the same three metric values across 4 picks. The matching short-sample summaries are descriptive; the package’s version-change entries report no changes for either model in the supplied comparison, but that does not prove their performance is stable.
Deepseek-v4.1-flash reports ROI of +54, Brier of 0.2372 and CLV of +0 on 16 picks. Its sample is the largest of the models listed, but still limited for conclusions about repeatable returns. Apodex-1.1-mini reports ROI of -24.1, Brier of 0.2379 and CLV of +0.3 on 10 picks; the positive CLV figure does not coincide with positive ROI in this sample.
Gpt-6-luna-pro pairs ROI of -26.5 with Brier 0.268 and CLV +3.8 across 5 picks. Nemotron-3.5-lightning reports ROI of -33, Brier 0.2695 and CLV +3.4 on 5 picks. Claude-haiku-5.5 has the same reported values for those three metrics and the same pick count. For all three, the small samples limit what can be inferred from the relationship between CLV, probability score and return.
What this means for readers
The benchmark shows that positive ROI, a lower Brier score and positive CLV do not line up uniformly across the reported models. In particular, the listed results include positive CLV alongside negative ROI, and a strong Brier score alongside negative ROI. Those are observations from this frozen sample, not proof that one metric is more useful in every setting or that any model will continue to perform similarly.
Readers can inspect individual results on the deepseek-v4.1-flash model profile and ling-3.1-flash model profile, then review metric definitions and Alpha Bench methodology. For context on interpreting return and model comparisons, see the related research on why high win rate does not always mean profitability. The practical takeaway is to read ROI alongside sample size, probability accuracy, CLV and observed drawdown; this benchmark does not supply a basis for guarantees or betting advice.
Methodology
Alpha Bench identifies the results as a frozen benchmark using methodology version 2.1. Its methodology page describes the evaluation framework, including how pre-kickoff snapshots and settled outcomes are used to assess model performance. Brier and log loss summarize probability error, CLV summarizes the comparison with closing prices, and drawdown describes the observed decline from a bankroll peak; these measures capture different dimensions rather than interchangeable definitions of success.
The package groups the results across the included pilot scope, but supplies no named pilot leagues and no market-specific performance rows. The prediction and odds-snapshot arrays are empty here, so readers cannot reproduce individual forecasts or prices from this article’s package alone. The methodology page and data sources page provide the internal references for checking how Alpha Bench defines and presents its evaluation data.
Limitations
The main limitation is sample depth: the overall sample is labeled Early sample, and model-specific settled-pick counts range from 2 to 16. The stated evaluation window extends through 2026-10-31, while the package cutoff is 2026-10-18T16:00:02Z. The comparison therefore cannot be treated as a final result for the full scheduled window.
A second limitation is scope: the package provides no pilot-league names, no model-level win rates and no market-specific performance rows. Calibration is aggregated rather than separated by model or market, and the supplied prediction and odds-snapshot arrays contain no entries. These gaps prevent conclusions about which model is best calibrated in a specific market, whether win-rate differences explain ROI, or whether the observed ordering generalizes beyond this sample.
Reproducible data references
This article uses the frozen run 2026-10-18-vector-001, data cutoff 2026-10-18T16:00:02Z and methodology version 2.1. Its relevant benchmark records are vector-dataset-status, vector-model-comparison, vector-long-term-patterns, vector-calibration, vector-risk-metrics, vector-market-performance-all, vector-league-performance-benchmark and vector-version-changes. The model-level table and notes use the supplied per-model settled-pick, Brier, ROI and CLV fields.
To check individual values, readers can follow the linked model profiles; for metric definitions and evaluation design, use the trust and metric definitions page and methodology page. Competition pages and fixture transparency blocks are not linked in this package, so they cannot be used here to verify league- or fixture-level claims. The limits of verification are part of the result: the frozen data supports a cautious cross-model comparison, not a reconstruction of every underlying pick.