Overconfidence Calibration for UX Researchers

Description

Elicits researchers' and users' 90% confidence intervals or probability forecasts, scores how often they contain the truth, and trains teams toward well-calibrated predictions.

Calibration measures whether stated confidence matches accuracy: a well-calibrated person's 90% confidence intervals contain the true value about 90% of the time, and events they rate 70% likely occur about 70% of the time. Most people, including experts, are overconfident, with nominal 90% intervals capturing the truth only 40-60% of the time. For UX teams the method has two uses. First, researchers and stakeholders record predictions before studies (for example task success rate or conversion lift) and are scored afterwards with hit rates or Brier scores, revealing systematic overconfidence and making the value of research visible. Second, user confidence ratings during tasks can be compared with actual correctness to locate interfaces that make people confidently wrong. Calibration improves measurably with feedback and practice.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Researchers provide 90% CI estimates for study outcomes.
  2. After study, compute hit rate.
  3. Calibration target: hit rate within ±10% of stated confidence level.
  4. Run as Discovery Series exercise.

Sources

  • Heuristics and Biases — Kahneman (ed.)
  • How to Measure Anything (Douglas Hubbard)
  • Superforecasting (Philip Tetlock & Dan Gardner)

Also known as

  • Calibration training
  • Confidence interval elicitation
  • Researcher prediction calibration

Categories

  • Research Operations
  • Research Design & Rigor