Abstract
This report summarises the September 2026 evaluation cycle of AlphaBench, covering 270 settled fixtures across five pilot leagues (Premier League, LaLiga, Bundesliga, Serie A, Champions League) from 2026-09-01 through 2026-09-30. Grok-build-0.1 recorded the strongest probabilistic accuracy of the four tracked models (Brier 0.2164, log loss 0.6237), while grok-4.3 posted the only positive return on settled picks (+21.21% ROI) on a much smaller sample of 17 picks. Across all populated probability bins, realised outcomes fell below predicted probabilities, indicating a systematic over-confidence pattern in this period's cohort.
Evaluation period
Settled fixtures included in this report fall within the UTC window 2026-09-01 through 2026-09-30. A total of 270 settled fixtures were counted across the five pilot leagues: Premier League, LaLiga, Bundesliga, Serie A, and Champions League. Only fixtures with a final settlement outcome and a pre-kickoff odds snapshot were eligible for scoring; the leaderboard filters applied here match those described below.
Executive summary
Grok-build-0.1 leads the September leaderboard on both headline probability metrics, with a Brier score of 0.2164 and log loss of 0.6237 across 117 settled picks. Gpt-5-mini is a close second on Brier (0.2247) but sits lower on log loss (0.6792) with 64 settled picks, suggesting occasional high-confidence errors rather than broadly weaker calibration. Deepseek-flash sits third on Brier (0.2847) yet has the least negative ROI of the top three (-8.01%, 91 picks), and grok-4.3 is fourth on Brier (0.2852) but the only model with positive ROI (+21.21%) — on just 17 settled picks, a sample far too small to treat as a stable signal.
The clearest structural takeaway is not which model "won" but how the cohort behaved in aggregate: in every populated probability bin, the realised hit rate came in below the predicted rate, from the 30-40% bin (predicted 0.3471, actual 0.0857, 35 picks) through the 80-90% bin (predicted 0.8325, actual 0.4615, 13 picks). Readers should treat ROI differences here as descriptive of one month, not as a ranking of model quality.
Leaderboard
| Rank | Model | Brier ↓ | Log loss ↓ | ROI | CLV | Settled picks | | --- | --- | --- | --- | --- | --- | --- | | 1 | grok-build-0.1 | 0.2164 | 0.6237 | -12% | — | 117 | | 2 | gpt-5-mini | 0.2247 | 0.6792 | -14.2% | — | 64 | | 3 | deepseek-flash | 0.2847 | 0.7732 | -8% | — | 91 | | 4 | grok-4.3 | 0.2852 | 0.7918 | +21.2% | — | 17 |
Calibration
Calibration is reported across nine populated probability bins. The pattern in September 2026 is directional and consistent: predicted probabilities exceed realised outcomes in every bin with a non-zero actual rate.
| Decile | Predicted | Actual | Count | |---|---|---|---| | 10-20% | 0.1433 | 0.0000 | 3 | | 20-30% | 0.2643 | 0.2000 | 15 | | 30-40% | 0.3471 | 0.0857 | 35 | | 40-50% | 0.4360 | 0.2000 | 45 | | 50-60% | 0.5477 | 0.3973 | 73 | | 60-70% | 0.6396 | 0.4462 | 65 | | 70-80% | 0.7268 | 0.6111 | 36 | | 80-90% | 0.8325 | 0.4615 | 13 | | 90-100% | 0.9701 | 0.5000 | 2 |
The largest bin by volume, 50-60% (73 picks), shows a gap of roughly 0.15 between predicted and actual, while the 40-50% bin (45 picks) shows a gap of about 0.24. High-confidence bins are thin: the 80-90% bin contains 13 observations and the 90-100% bin only 2, so those apparent shortfalls carry very wide uncertainty and should not be interpreted as a stable property of any single model. No bin below 10% is reported, and no league-level decomposition is included in this month's published set.
Taken together with the Brier and log loss values in the leaderboard, this pattern is consistent with a cohort-wide tendency to assign more probability to outcomes than September's results supported. Log loss is more sensitive to confident mistakes than Brier, which is one plausible reason the two metrics order the middle of the table differently; this report does not establish that explanation.
CLV
Closing-line value (CLV) was not populated for any model in the September cohort; all four leaderboard entries carry a null CLV value. CLV is defined here as the difference between the price at which a pre-kickoff pick was recorded and the closing price from the designated reference bookmaker snapshot, measured in the same market and line. Because no CLV observations were available for 2026-09-01 through 2026-09-30, no average CLV by model, market type, or pilot league can be reported, and no CLV-based conclusions should be drawn from this cycle. ROI figures in the leaderboard should likewise be read without CLV corroboration.
Limitations
Several constraints bound what these numbers can support. First, scope: results cover five pilot leagues only, and are not generalisable to other competitions, divisions, or markets. Second, sample size: settled pick counts range from 17 (grok-4.3) to 117 (grok-build-0.1), and models with small counts have correspondingly unstable Brier, log loss, and ROI values — the positive ROI of 21.21% rests on 17 settled picks. Third, calibration bins at the extremes are sparse, with the 80-90% bin holding 13 observations and the 90-100% bin holding 2. Fourth, CLV is entirely missing for this period, so no market-efficiency cross-check is available. Fifth, a single month is not a trend, and month-to-month movement should be expected from variance alone. This report is educational research on model evaluation; it is not betting advice, and no figure here should be used as a basis for wagering decisions.
Methodology note
Scoring rules, immutability guarantees, and metric definitions are documented in /blogs/methodology and /blogs/model-evaluation, with a plain-language walkthrough of scoring in /blogs/how-alpha-bench-scores-models. In brief, predictions are frozen before kickoff and are never revised after the fact; settlement follows the rules described in /blogs/settlement-rules; and input provenance is described in /blogs/data-sources. Brier score and log loss are computed over the same settled probability set for each model, and ROI is computed over settled picks only.
Data cutoff
All metrics in this report were frozen for publication at 2026-10-01 07:30 UTC. The primary odds snapshot used for evaluation was captured pre-kickoff for each fixture, and only fixtures settled within the 2026-09-01 to 2026-09-30 window with a valid pre-kickoff snapshot were included. Fixtures settled after the cutoff, or lacking a qualifying snapshot, are excluded from the counts shown here and may appear in a later cycle. The current standings are also available on the /leaderboard.