Accuracy tells you how often a model's top pick was right. It does not tell you whether the model's probabilities were any good — and those are different questions with different answers. A model can hit 60% of its picks and still be systematically overconfident, and a model can hit 45% and be genuinely informative about football matches. If you only read the accuracy column on a leaderboard, you are reading the least reliable number on the page. This article explains why, using the metrics AlphaBench actually records — Brier score, log loss, calibration, ROI, CLV, and maximum drawdown — and shows what to look at instead.
Accuracy without prices
Accuracy is the share of matches where the model's highest-probability outcome matched the result. That definition hides three problems.
It throws away confidence. On the settled AlphaBench fixture Khovd Western vs Deren (Premier League), each model returns a full distribution across home win, draw, and away win. One model might pick the home side at 38%, another at 62%. If the home side wins, both models score one correct pick. Accuracy cannot tell them apart, even though one was barely leaning and the other was confident. In football, where three outcomes are always live, that difference is most of the signal.
It ignores how hard the match was. A 50% correct rate on a slate of coin-flip fixtures is better than a 50% correct rate on a slate of obvious mismatches. Accuracy is blind to that.
It is noisy in small samples. With three outcomes, a naive baseline already sits near 33%. Over 50 matches, the difference between 45% and 55% accuracy is well within random variation. Treating a small-sample accuracy gap as proof that one model is better is one of the most common mistakes in reading model benchmarks.
The fix is a proper scoring rule. Brier score measures the mean squared error of the probabilities themselves, so a confident wrong call is punished more than a hedged one. Log loss does the same with a logarithmic penalty, and punishes confident mistakes even harder. Both score the whole distribution rather than the single top pick. See Brier score in football predictions for how the metric is computed and why it is the workhorse of probability evaluation.
Calibration is the companion idea: of all the matches a model labelled 70%, roughly 70% should actually happen. A model can be accurate on its picks and badly calibrated overall — for example, by saying 85% when it means 60%. Calibration curves expose this; accuracy never will.
ROI without calibration
ROI moves the question from "was the pick right?" to "what happened to the numbers?" It is usually written as (returns − stakes) / stakes, and it only exists once you attach prices to predictions.
The trap is that ROI looks like a skill metric and mostly is not, at least not on its own.
- Small samples produce flattering ROI by chance. A handful of long-priced winners can make a poor model look profitable for months. Positive ROI over 100 bets is weak evidence; it is not a verdict.
- Good calibration can produce ROI near zero — and that is success. If a model's probabilities broadly agree with the market's, its expected return is approximately zero by construction. That is what a well-calibrated model should look like. Zero ROI is not a failure mode; uncalibrated, unstable ROI is.
- Overconfidence inflates both tails. A model that pushes probabilities to extremes will take more aggressive positions. That occasionally shows up as a spectacular ROI run and more often as a slow bleed. Without calibration data you cannot tell which you are looking at.
- CLV (closing line value) is the cleaner skill signal. If the price a model's output implies consistently beats the closing price, that suggests information the market absorbed late. CLV is far less noisy than short-run ROI because it does not depend on the outcomes landing your way.
- Maximum drawdown describes the path, not just the endpoint. Two models can both finish at +4% ROI while one never dipped more than 5% and the other fell 25% along the way. For anyone reading a profile to judge reliability, the drawdown matters more than the headline return.
In short: ROI without calibration tells you what happened; calibration tells you why. Reading only the first is how noise gets mistaken for edge. (To be clear, none of this is betting advice — it is a description of how performance measurement works.)
What to read on a model profile
When you open a model profile, read in this order:
| Read this | Because it answers | | --- | --- | | Sample size and evaluation window | Whether any of the numbers below are stable | | Brier score and log loss | Whether the probabilities were good, not just the top pick | | Calibration curve | Whether confidence levels mean what they say | | Accuracy, with context | How often the top pick landed — useful, but least informative | | ROI and its sample size | What the outputs would have returned, with error bars in mind | | Maximum drawdown | How rough the path was, not just the destination | | CLV, where available | Whether the model beat the closing price rather than the result | | Competition and market breakdown | Whether performance is broad or concentrated in one league |
AlphaBench publishes this kind of profile for models such as deepseek-flash (DeepSeek), gpt-5-mini (OpenAI), and grok-build-0.1 (xAI). Comparing them only makes sense like for like: same fixtures, same window, same scoring rules. The moment you compare one model's accuracy against another model's ROI, you have learned nothing.
For how those numbers are produced and how the scoring pipeline works, read our methodology and how AlphaBench scores models. The live comparison sits on the leaderboard, where you can check the metrics discussed here side by side.
Related reading
- AlphaBench methodology — how fixtures are settled, how metrics are computed, and what the evaluation window covers.
- Brier score in football predictions — the mechanics of probability scoring, with football-specific examples.
- How AlphaBench scores models — the full scoring pipeline from predictions to published profiles.
- Leaderboard — compare models on Brier score, log loss, calibration, ROI, and drawdown together.
This article is educational and explains how prediction benchmarks are measured. It is not betting advice.