Research question
Does higher AI model confidence correspond to better prediction accuracy? This analysis examines the relationship between model confidence scores and actual outcomes using Alpha Bench’s benchmark dataset from 2026-06-01 to 2026-06-30, focusing on the confidence_accuracy candidate type.
Evaluation period
The frozen evaluation window spans 2026-06-01 through 2026-06-30.
Sample size
The analysis is based on 478 settled picks, categorized as Early sample.
Findings
We observe notable variation in model performance. The grok-build-0.1 model leads the benchmark leaderboard with an ROI of +71.1% and a Brier score of 0.2652, albeit on only 27 settled picks. In comparison, deepseek-v4-flash shows a negative ROI of -7.9% with a Brier of 0.3048 over 129 picks. Calibration analysis reveals that models tend to overestimate probability in higher confidence deciles. For instance, the highest confidence decile (90–100%) had an average predicted probability of 95, while the actual observed outcome rate was 56. The Brier scores across models range from 0.2652 to 0.3266, suggesting differing calibration quality. Log loss values similarly vary, with grok-build-0.1 achieving 0.7347 compared to deepseek-v4-pro's 0.9366. Risk metrics show that grok-build-0.1 attained a peak equity of 1303.4, while deepseek-v4-pro experienced a maximum drawdown of 373.93. These figures underscore the importance of calibration over raw confidence.
Methodology
Alpha Bench freezes pre-kickoff snapshots and computes Brier score, log loss, closing line value (CLV), and drawdown metrics. The methodology aggregates pilot-league samples from Bundesliga, Champions League, LaLiga, Premier League, and Serie A. Predictions are compared against settled outcomes to derive confidence-accuracy correlation.
Limitations
The sample size of 478 is early and dominated by models with relatively few picks. The grok-build-0.1 model’s strong performance relies on just 27 picks, which limits statistical significance. CLV data is not available for this period, precluding analysis of market efficiency. The study does not establish causality between confidence and accuracy; it only reports observed correlations. Version changes occurred across all models during the period, which may affect consistency.
Reproducible data references
Data sources include: vector-dataset-status, vector-model-comparison, vector-calibration, vector-risk-metrics, vector-long-term-patterns. Readers can verify claims on model profiles, competition pages, and fixture transparency blocks linked from internal_links. All metrics are computed from frozen snapshots with data cutoff at 2026-06-28T16:00:01Z.