Probability inflation ratio

Description

Quantifies how much stakeholders or users overestimate the probability of a behaviour by dividing predicted probability by observed frequency.

Teams routinely overestimate how often users will adopt features, complete flows or use advanced functions, through availability, representativeness and optimism biases. The probability inflation ratio makes the bias measurable: collect explicit probability forecasts before data exists, then divide the forecast by the observed rate. A ratio of 1.0 is perfectly calibrated; 3.0 means the event was predicted three times as often as it happened. Tracked across forecasts and forecasters, it shows systematic bias by person, feature type or user segment, and can be used to discount future estimates. Rigour comes from recording forecasts in writing in advance, defining the behaviour and denominator precisely, and pairing the ratio with a proper scoring rule such as the Brier score.

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. Define the behaviours to forecast with an exact numerator, denominator and time window, for example 'share of active health workers who use the referral feature at least once in 30 days'.
  2. Before launch or before data is visible, collect written probability forecasts from each stakeholder or participant, ideally a central estimate plus an 80% interval.
  3. After the window closes, compute the observed rate from logs and the inflation ratio as forecast divided by observed for each forecaster and behaviour.
  4. Aggregate ratios using the geometric mean (ratios are multiplicative) and compute Brier scores to separate calibration from discrimination.
  5. Break results down by forecaster role and behaviour type (basic versus advanced features, new versus habitual behaviours) to find systematic inflation patterns.
  6. Feed back results to forecasters and apply the historical ratio as a correction factor in roadmap business cases, re-estimating every 2 to 4 forecasting cycles.

Sources

  • Heuristics & Biases (Kahneman ed.)
  • Judgment Under Uncertainty
  • Choices Values & Frames (Kahneman/Tversky)
  • The Inquiry Series, Ch 21 — Heuristics, Biases & the Architecture of Judgment

Also known as

  • Forecast inflation ratio
  • Predicted-to-observed ratio
  • Stakeholder calibration ratio

Categories

  • Analytics & Behaviour
  • Behavioral Science & Economics