A probability is only useful if it means what it says. If a model says "62% chance the home team wins" across a hundred matches, roughly sixty-two of those matches should end in a home win — that agreement between stated confidence and observed frequency is what calibration measures, and it is the single most misunderstood property in football forecasting. A model can be brilliant at ranking fixtures (picking the right favourite) and still be badly calibrated (saying 90% when it means 70%). AlphaBench reports calibration separately from accuracy for exactly this reason, and the methodology page explains how that split shapes every score on the board. This article covers what calibration is, how reliability diagrams expose it, why it is not the same thing as discrimination, and what a sensible pilot report should contain.
Calibration is a property of the probabilities a model emits, not of the model's ordering of outcomes. To measure it, you collect many predictions, group them into buckets, and compare the average predicted probability in each bucket with the actual hit rate. Because football produces three-way outcomes (home, draw, away), calibration is normally assessed per outcome class or on the full probability vector, not just on the favourite. That detail matters: a model can look well calibrated on home wins and be systematically overconfident on draws, which is one of the hardest classes in the sport.
Reliability diagrams
A reliability diagram — sometimes called a calibration curve — is the standard visual test. The x-axis is the predicted probability the model assigned; the y-axis is the observed frequency of that outcome in the same bucket. Predictions are sorted into bins (typically ten equal-width or equal-count bins), each bin gets a point, and a perfectly calibrated model traces the 45-degree diagonal: predict 30%, observe 30%.
Two failure modes show up immediately. Points above the diagonal mean the model is underconfident — it said 30% and the outcome happened 45% of the time. Points below the diagonal mean overconfidence, the more common and more damaging failure: the model's "certainties" are not as certain as advertised.
Take a concrete settled fixture such as Khovd Western against Deren in the Premier League. In isolation, one result tells you nothing about calibration — Khovd Western either won the match or did not. Calibration is a property that only emerges across a population of such fixtures. If a model assigned the home side between 0.60 and 0.70 in two hundred comparable matches, and the home side actually won about 130 of them, that bin sits close to the diagonal. If it won only 100, the model is overconfident in that band.
Reliability diagrams are also useful for spotting bin-level noise. With small samples, individual bins jump around, so benchmarks typically report confidence intervals or switch to equal-count bins so each point rests on the same number of predictions. A reliability diagram with three matches per bin is a sketch, not evidence.
Calibration vs discrimination
These two ideas are often conflated, and separating them is the core skill this article is trying to teach.
Discrimination is the ability to rank outcomes correctly — to give higher probabilities to things that actually happen than to things that do not. It is measured by metrics like the area under the ROC curve or by the separation between the score distributions of winners and losers.
Calibration is the ability to attach the right magnitude to those rankings. It is measured by reliability diagrams, the Brier score, and log loss.
A model can be high in one and low in the other. Imagine a model that always ranks the true favourite first but outputs 0.95 for every favourite. Its discrimination may be excellent while its calibration is poor, because it is claiming far more certainty than it has earned. The reverse also happens: a model that outputs sensible-looking frequencies (roughly 45% of home-win predictions come true) but orders fixtures almost at random.
The two headline metrics on AlphaBench's scoring page capture this split. The Brier score is a mean squared error over probabilities, so it rewards both good ranking and honest magnitudes — a deeper treatment lives in the Brier score explainer. Log loss punishes confident mistakes far more sharply, so a model that says 0.97 and is wrong is penalised brutally. Neither metric alone tells you which property failed, which is why the reliability diagram and the discrimination curve are reported alongside them.
A practical illustration: suppose deepseek-flash, gpt-5-mini, and grok-build-0.1 all agree on the ordering of a matchday's fixtures but disagree on magnitudes. Their discrimination scores could be near-identical while their Brier and log loss diverge sharply, and the reliability diagram would show which of them is overconfident in the 0.70–0.90 bands. That is the diagnostic pattern benchmark readers should learn to look for.
Pilot reporting
A pilot is a small, honest trial run — and its purpose is to find out whether a model's probabilities behave, not to declare a winner. The following checklist covers what a defensible pilot report contains.
State the sample size and window. Calibration estimates are unstable on small samples. A pilot over fifty matches can hint at a problem but cannot settle one. Report the number of predictions per bin, and prefer equal-count bins so no point rests on a handful of matches.
Report per-class calibration. Football is three-way. Give reliability diagrams or calibration tables for home, draw, and away separately. Draws are where most models misbehave, and averaging them into one number hides it.
Report the proper scoring rules. The Brier score and log loss belong in every pilot, reported with a baseline for comparison — a naive base-rate model that always predicts the historical league frequencies is the standard yardstick. Metrics without a baseline are decoration.
Report the diagram, not just the number. A single calibration error statistic compresses a curve into a scalar and can hide offsetting errors in different probability bands.
Separate calibration from discrimination. Include a discrimination measure so readers can tell whether a weak Brier score came from bad ordering or bad magnitudes.
Include market-relative and bankroll-style diagnostics with care. CLV (closing line value) measures whether a model's prices beat the closing market, and ROI plus maximum drawdown describe the shape of a hypothetical staking curve — including how deep the worst peak-to-trough decline went. These are useful stress tests of a probability model, but they are descriptive statistics from a pilot, not forecasts. Small-sample ROI is dominated by variance, and maximum drawdown is extremely sensitive to the length of the window.
Report the caveats. Leakage, changing league strength, mid-season managerial changes, and fixture congestion all shift the underlying distribution. A model calibrated on last season's data may be miscalibrated this season, which is why rolling evaluation windows matter.
Do not over-interpret. A pilot is a diagnostic. The honest conclusion is usually "the model is overconfident between 0.7 and 0.9 on away wins, and we need more data," not "this model is best."
None of this is betting advice, and none of it should be read as a staking recommendation. Calibration is a measurement question: does the number mean what it claims?
Related reading
Start with the AlphaBench methodology for how predictions are collected, timestamped, and evaluated. The Brier score explainer unpacks the most common proper scoring rule in detail. The scoring guide covers how calibration, discrimination, and market-relative measures are combined. And when you want to see these properties measured across models on real settled fixtures — including matches like Khovd Western versus Deren — the leaderboard is the place to look.