Brier Score and Calibration Analysis

Description

Scores probabilistic predictions against outcomes and decomposes accuracy into calibration and resolution, applicable to researchers, users and AI systems.

The Brier score is the mean squared difference between forecast probabilities and outcomes (0 or 1); lower is better. Murphy's decomposition separates reliability (calibration: do 70% forecasts happen 70% of the time?), resolution (do forecasts discriminate?) and uncertainty. Calibration curves plot predicted against observed frequency. Weather forecasters became well calibrated through feedback. In UX this applies three ways: calibrating researchers' and teams' predictions about tests and launches; evaluating AI confidence displays shown to users; and measuring whether users' own confidence judgements are calibrated.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Hours

Steps

  1. Collect probabilistic forecasts (e.g. teams predict the probability each A/B test wins, or an AI model gives confidence scores).
  2. Record outcomes once known.
  3. Compute Brier scores and Murphy decomposition; plot calibration curves with binned or smoothed estimates.
  4. Compare forecasters or models, and against base-rate forecasts (Brier skill score).
  5. Feed results back to forecasters regularly to improve calibration.
  6. For AI products, recalibrate model confidence (e.g. temperature scaling) before displaying it to users.

Sources

  • Brier, Verification of forecasts expressed in terms of probability (1950), Monthly Weather Review
  • Murphy, A new vector partition of the probability score (1973), Journal of Applied Meteorology
  • Tetlock & Gardner, Superforecasting (2015)

Also known as

  • Brier score
  • Calibration curves
  • Forecast verification
  • Proper scoring rules

Categories

  • Strategy & Collaboration
  • Game Theory & Decision Science