AlphaBench scores a football model in one direction only: it measures how good the model's probabilities are, not how entertaining or confident they sound. The takeaway for a reader is simple — a model earns its place on the leaderboard by producing well-calibrated, properly-scored forecasts over a fixed, rolling window of settled fixtures, with the scoring rules published in advance and applied identically to every entry, whether that entry is deepseek-flash, gpt-5-mini, grok-build-0.1, or anything else. Everything below explains how that works, what the numbers mean, and where the guardrails sit.
Primary metrics
AlphaBench uses a layered metric stack. Proper scoring rules come first, because they are the only measures that reward a model for stating honest probabilities rather than hedging or grandstanding.
Brier score. The Brier score is the mean squared error between a forecast probability and the actual outcome. For a three-way football market (home, draw, away) it is computed as the sum of squared differences across the three probabilities, averaged over fixtures. Lower is better. A model that says 90% and is right gains less than it loses when it says 90% and is wrong, which is exactly the incentive you want. Our dedicated walkthrough of the maths lives at /blogs/brier-score-football-predictions.
Log loss. Log loss (cross-entropy) applies a logarithmic penalty to the probability assigned to the event that actually occurred. It punishes confident mistakes far harder than the Brier score does, so the two metrics together distinguish a model that is slightly overconfident from one that is recklessly so. Lower is better here too.
Calibration. A metric can flatter a model that never commits. Calibration checks the other direction: when a model says 30%, does the outcome happen roughly 30% of the time? AlphaBench reads calibration through reliability curves and a calibration slope, so a model that is directionally good but systematically over-pitched shows up as a visible defect rather than a hidden one.
CLV (closing line value). CLV compares the price implied by a model's forecast against the closing market price for the same selection. It is a market-relative measure, and it is reported separately from the probability metrics precisely because it answers a different question: not "were the probabilities good?" but "did the model's view tend to sit on the right side of where the market finished?"
ROI and maximum drawdown. ROI expresses return over a defined staking convention. Maximum drawdown measures the deepest peak-to-trough decline in that same equity curve. Both are included because a headline ROI without a drawdown figure hides how uncomfortable the path was. Neither is a promise of anything, and neither is betting advice.
The important design choice is order. Probability metrics (Brier, log loss) and calibration carry the most weight because they are hard to game and are computed from the forecast itself. Market-relative and bankroll-relative metrics (CLV, ROI, maximum drawdown) are reported alongside them as context. A single settled fixture — for example Khovd Western vs Deren in the Premier League — contributes exactly one row to all of these calculations once it settles, no more and no less.
Evaluation window
Scores are not computed over "all history" and they are not computed over a hand-picked stretch. They are computed over a declared window.
- Settled fixtures only. A fixture enters the window when it has a final result. Open or abandoned fixtures are excluded rather than imputed.
- Rolling rather than cumulative. The headline figures typically reflect a trailing window — season-to-date and a trailing block of fixtures — so that a model cannot coast on a strong month from long ago, and a new model is not permanently buried by a small early sample.
- Walk-forward, no lookahead. Forecasts are scored against the state of the world at the time they were made. A model cannot be credited for information that only became available after kickoff.
- Sample size is disclosed. A window with a handful of fixtures is noisy, and AlphaBench shows how many settled fixtures sit behind a number. One match settles one row; a meaningful gap between two models needs many rows.
- Windows move, scores move. Because the window rolls, a model's rank can shift even when the model itself has not changed. That is a property of the window, not a bug, and it is why the leaderboard is read as a current snapshot rather than a verdict.
Governance
Scoring is only credible if the rules are fixed before the results arrive. AlphaBench's governance layer exists to make that true and checkable.
- Published methodology. The metric definitions, weighting, and window rules are documented up front at /blogs/methodology, so a model builder can know exactly what they are being measured on.
- Identical treatment.
deepseek-flash(DeepSeek),gpt-5-mini(OpenAI), andgrok-build-0.1(xAI) run through the same pipeline as every other entry. There is no bespoke metric, no manual adjustment, and no friendly re-run. - Versioning and disclosure. When a model changes materially, the change is recorded so that pre-change and post-change performance are not silently blended into one flattering line.
- No cherry-picking. Windows, fixtures, and exclusions are defined by rule rather than chosen after seeing the outcome. Retired or withdrawn models leave the board rather than lingering with frozen numbers.
- Separation of scoring and editorial. The scoring pipeline produces numbers; the writing around it explains them. Editorial interest does not feed back into the metric.
- Errors are handled in the open. Data corrections and disputed settlements are logged, because a benchmark that quietly edits its own history is not a benchmark.
Read together, these constraints mean an AlphaBench score is a statement about measured forecasting behaviour under fixed rules — not a prediction, not a tip, and not betting advice.
Related reading
- AlphaBench methodology — the full definition of metrics, weights, and windows.
- The Brier score in football predictions — a worked explanation of the primary metric.
- How AlphaBench scores models — the canonical reference version of this article.
- The AlphaBench leaderboard — current standings, with sample sizes shown.