Hindsight-Bias Measurement by Recalled Predictions

Description

Record people's predictions before an outcome, ask them to recall those predictions afterwards, and measure how far recall drifts towards what actually happened.

After an outcome is known, people remember having expected it more than they did. Fischhoff demonstrated this 'knew it all along' effect in the lab, and Roese and Maniar measured it in the field by recording football fans' predictions before games and asking for recall on the way home: fans remembered being more confident of surprising wins than they had been. Asking some fans to imagine how the game could have gone otherwise made the sense of inevitability stronger, not weaker. For research teams the measurement matters because hindsight inflates the perceived obviousness of findings, the apparent predictability of launch results and the blame attached to past decisions. Recording predictions in advance makes the bias measurable and gives a baseline that memory cannot rewrite.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Before an outcome (study result, experiment readout, launch), collect concrete predictions as probabilities or outcome bands from each person.
  2. Store the predictions where participants cannot consult them.
  3. After the outcome, ask the same people to recall their earlier predictions.
  4. Compute each person's shift from recorded to recalled prediction, signed towards the actual outcome, and the group mean.
  5. Optionally assign some people to a prompt (consider alternative outcomes, review recorded predictions) before recall and compare shifts.
  6. Share recorded predictions alongside results to calibrate how surprising findings really were.

Sources

  • On Second Thought: Outsmarting Your Mind's Hard-Wired Habits (Herbert, 2010)

Also known as

  • Knew-it-all-along test
  • Recalled versus recorded prediction comparison
  • Counterfactual debiasing test

Categories

  • Research Operations
  • Research Design & Rigor