A Brier score turns a football model's probabilities into one number: it measures the squared distance between what the model said would happen and what actually happened. Lower is better, 0 is perfect, and a 1X2 model that cannot beat the 0.667 you get by saying "one third, one third, one third" on every fixture is not telling you anything useful. This article explains what that number means, how AlphaBench computes it on settled fixtures, and how to read it honestly on a pilot leaderboard — without turning any of it into a betting tip.
Why probability quality matters
Football models do not really output picks. They output probability vectors: home win, draw, away win. A vector like 0.55 / 0.25 / 0.20 is a claim about the world, and it can be evaluated the moment the fixture settles. That is the whole point of the exercise — a forecast is only as good as its willingness to be graded.
Two ideas do most of the work here:
- Calibration. If a model says 70% many times over a season, roughly 70% of those cases should resolve that way. Calibration is about honesty, not cleverness.
- Discrimination (resolution). A well-calibrated model that always says 0.45 / 0.27 / 0.28 is honest but useless. Good models separate fixtures that are genuinely different — a lopsided mismatch should not look like a coin flip.
Accuracy alone cannot see either property. Predicting the single most likely outcome and ignoring the rest throws away the probabilities that make a model interesting, and it rewards a model that is confidently wrong in exactly the same way as one that was nearly right.
The Brier score is a proper scoring rule, which means it is minimised by reporting your true beliefs. Overstate a probability to look bold and the score punishes you; hedge everything toward the middle and the score punishes that too. Log loss is the close cousin of Brier and behaves the same way, but with an unbounded penalty for confident mistakes — a model that assigns 2% to the outcome that happens gets destroyed by log loss while surviving a Brier score. AlphaBench reports both because they stress different failure modes. Other metrics in the vocabulary — CLV, ROI, maximum drawdown — belong to the market-facing side of evaluation; they depend on prices and staking, whereas Brier and log loss grade the model itself. None of this is betting advice; it is forecast evaluation.
How AlphaBench computes Brier score
For a three-way football market the multiclass Brier score over N settled fixtures is:
`` BS = (1/N) * Σ_i Σ_k (p_ik − o_ik)² ``
where p_ik is the model's probability for outcome k in fixture i, and o_ik is 1 if that outcome occurred and 0 otherwise.
In practice, once a fixture settles, the result becomes a one-hot vector — (1, 0, 0) for a home win, (0, 1, 0) for a draw, (0, 0, 1) for an away win. Take a settled fixture such as Khovd Western vs Deren in the Premier League. Suppose (illustratively, not an actual logged forecast) a model posted 0.55 / 0.25 / 0.20:
- If Khovd Western win: (0.55 − 1)² + (0.25)² + (0.20)² = 0.305
- If Deren win: (0.55)² + (0.25)² + (0.20 − 1)² = 1.005
Same forecast, wildly different penalty depending on the result. That asymmetry is intentional: it is what forces the model to spread probability where uncertainty actually exists.
A few reference points make the scale readable:
| Forecast behaviour | Brier score (1X2) | | --- | --- | | Perfect (probability 1 on the actual outcome every time) | 0.000 | | Uniform 1/3 on every fixture | 0.667 | | Worst possible (probability 1 on a wrong outcome) | 2.000 |
The 0.667 line is the important one. It is the "no information" baseline, and it is the bar any model has to clear before its Brier score means anything.
Two procedural details matter as much as the formula. First, AlphaBench scores every settled fixture in a fixed evaluation window, not a curated subset — cherry-picking easy fixtures would flatter the number. Second, models are compared on the same fixtures so the scores are directly comparable. The full scoring pipeline, including how fixtures are selected and how logs are frozen before kickoff, is described in the AlphaBench methodology and in more detail in how AlphaBench scores models.
Interpreting pilot leaderboard values
Pilot leaderboards are small, and that shapes how you should read them.
Lower is better, and the baseline is the anchor. A model sitting at 0.66 has barely separated itself from uniform guessing. Models that land clearly below 0.667 are extracting real signal about the match. In a sport as high-variance as football, the honest range of good 1X2 Brier scores is narrow, so treat small gaps with respect.
Small gaps are usually noise. The standard error of an average Brier score shrinks with the square root of the number of settled fixtures. Over a few dozen fixtures, a 0.01–0.02 gap between two models is well within sampling error; only a consistent gap across a growing sample is evidence. The pilot leaderboard is a directional signal, not a verdict.
Equal Brier scores can hide different models. Two entries can land on the same number with one being overconfident on favourites and the other being timid across the board. Pull the calibration view before drawing conclusions: Brier compresses calibration and discrimination into one figure, and the full leaderboard is where the per-model breakdown lives.
Different model families, same scoring rule. The pilot includes entries from several providers — for example deepseek-flash from DeepSeek, gpt-5-mini from OpenAI, and grok-build-0.1 from xAI. They are graded identically, so provider or architecture is irrelevant to the number itself; only the logged probabilities count. Any apparent ranking you infer from prose (including this article) is provisional until you check the live table.
Do not translate Brier into stakes. A good Brier score says the probabilities are coherent, not that any particular fixture is worth acting on. Market-relative measures such as CLV, ROI and maximum drawdown answer a different question and depend on prices you may never get. Keep the two conversations separate.
A practical reading order for the pilot table: check the baseline first, check the sample size second, check calibration third, and only then compare model names. For the underlying theory and the arithmetic, see this article's companion primer on Brier score in football predictions.
Related reading
- AlphaBench methodology — how fixtures are selected, logged and scored.
- How AlphaBench scores models — the full metric stack, including Brier score and log loss.
- What Brier Score Means in Football Predictions — the canonical version of this explainer.
- The AlphaBench leaderboard — current pilot standings, refreshed as fixtures settle.
This article is educational and describes forecast evaluation only. It is not betting advice, and no metric described here should be read as a recommendation to place a wager.