Cognitive Bias Audit

Description

Systematically reviews a product's flows and the team's own decisions for heuristics and biases, distinguishing biases exploited in users from biases embedded by designers.

A cognitive bias audit applies the heuristics-and-biases programme as a structured inspection. It has two faces. Facing outward, it walks each user decision point and asks which judgement heuristics (representativeness, availability, anchoring, conjunction errors, framing) are likely operating, and whether the design mitigates, exploits or ignores them. Facing inward, it examines the team's own assumptions, for example through a prediction gap test in which stakeholders predict user behaviour before data is revealed, exposing overconfidence and availability effects in product decisions. Rigour comes from using an explicit bias checklist with definitions, recording evidence for each finding, rating severity, and validating predictions against behavioural data. Without this structure, 'bias' becomes rhetorical; with it, the audit becomes repeatable and comparable across releases.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Assemble a bias checklist with operational definitions (e.g. anchoring, availability, representativeness, conjunction fallacy, framing, default effect).
  2. Walk each decision point in the user journey and record which biases plausibly operate and whether the design mitigates or exploits them.
  3. Run a prediction gap test: have 5-10 stakeholders predict key user metrics before revealing actual data.
  4. Quantify prediction errors and identify systematic team biases such as overestimating advanced feature use.
  5. Rate each finding by severity and evidence strength, separating ethical risks from optimisation opportunities.
  6. Recommend mitigations and validate the highest-priority ones with behavioural data or experiments.

Sources

  • Heuristics and Biases
  • Thinking Fast and Slow
  • Heuristics & Biases (Kahneman ed.)
  • Judgment Under Uncertainty
  • Choices Values & Frames (Kahneman/Tversky)
  • Thinking, Fast and Slow (Daniel Kahneman)
  • The Inquiry Series, Ch 21 — Heuristics, Biases & the Architecture of Judgment

Also known as

  • Bias audit
  • Heuristics and biases review
  • Prediction gap test

Categories

  • Analytics & Behaviour
  • Behavioral Science & Economics