Actuarial versus clinical prediction

Description

Compare expert holistic judgement with a simple statistical rule built on the same cues, including equal-weight rules, and adopt whichever predicts outcomes more accurately.

In clinical prediction a judge combines information in their head; in actuarial prediction a rule combines it mechanically. Across about a hundred comparative studies the rule equalled or beat experts, and a meta-analysis of 136 comparisons found it at least as good in all but eight. Goldberg's simple MMPI rule was 70 per cent correct on 861 new cases, beating all 29 judges, and thousands of practice judgements with feedback did not close the gap. Dawes showed that improper linear models, with equal or even random positive weights on expert-chosen variables, matched or beat judges and fitted regressions and transferred better across settings. The expert's value lies in choosing variables, not combining them. Rules must be cross-validated, monitored and revalidated in each new setting.

Attributes

Loss type
Unmigrated
Effort
Weeks

Steps

  1. Define the decision, the criterion outcome and a set of past cases with known outcomes.
  2. Elicit from experts the cues they consider relevant.
  3. Collect experts' holistic predictions for a sample of cases.
  4. Build a simple rule on the same cues: a fitted regression, an equal-weight sum of standardised cues, or a tally.
  5. Cross-validate the rule on fresh cases not used to build it.
  6. Compare rule and expert accuracy on the same cases.
  7. Deploy the better approach and monitor accuracy, revalidating whenever the setting or population changes.

Sources

  • Heuristics and Biases (Gilovich et al., 2002)
  • The Alignment Problem (Christian, 2020)

Also known as

  • Statistical prediction rule
  • Mechanical prediction
  • Clinical versus actuarial comparison
  • Clinical versus statistical prediction
  • Improper linear models
  • Equal-weight models
  • Dawes tallying

Categories

  • Strategy & Collaboration
  • Game Theory & Decision Science