Analyst Debiasing Procedures

Description

Protect evaluators' judgements by blinding them to contaminants first, announcing accountability before evidence is seen, and prompting a few reasons they could be wrong.

Researchers rating severity, coding data or judging designs are vulnerable to unwanted influences they cannot detect. Wilson and Brekke's model orders defences: exposure control, preparation, resistance, remediation and behaviour control. People over-trust resistance and remediation, which usually fail because contamination is invisible and its size unknown, so blinding comes first. When blinding is impossible, rate items twice under different circumstances and re-examine disagreements, and check whether judgements covary with a suspected contaminant. Tetlock's work adds pre-decisional accountability: telling analysts before they see evidence that they must justify conclusions to an informed audience whose views are unknown prompts self-critical thinking and reduces primacy, anchoring and overconfidence, though it amplifies dilution and safe-option choices. Consider-the-opposite prompts, asking for a small number of reasons a judgement might be wrong, reduce overconfidence; asking for many backfires.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. List potential contaminants of the judgement (participant identity, brand, segment, condition, order, the stakeholder's preferred answer).
  2. Blind evaluators to these wherever feasible, for example by masking condition labels during coding or severity rating.
  3. Before evidence is seen, tell analysts they will justify conclusions to an informed reviewer whose views are unknown; do not reveal stakeholders' preferred conclusions.
  4. Where blinding is impossible, rate each item twice at different times or settings and re-examine items whose ratings differ.
  5. Compare ratings across subgroups defined by a suspected contaminant to detect covariation.
  6. After initial judgements, ask for two or three reasons each could be wrong, then re-elicit the judgement; avoid asking for many reasons.

Sources

  • Heuristics and Biases (Gilovich et al., 2002)
  • Mental Contamination and Mental Correction (Wilson & Brekke, 1994)
  • Accounting for the Effects of Accountability (Lerner & Tetlock, 1999)
  • Behavioral Finance and Decision Theory in Investment Management (Wood, 1995)

Also known as

  • Debiasing ladder for mental contamination
  • Exposure control
  • Blind evaluation
  • Pre-decisional accountability
  • Preemptive self-criticism
  • Consider-the-opposite prompts
  • Counter-reason listing

Categories

  • Research Operations
  • Research Design & Rigor