Structured Expert Judgement with Performance-Based Weighting (Cooke's Classical Model)

Description

Combine experts' uncertainty estimates using weights earned on seed questions with known answers, rewarding experts who are well calibrated and informative.

When data are missing, decisions lean on expert estimates, but experts differ in how well their uncertainty matches reality. Cooke's classical model makes this testable. Each expert is elicited individually on target quantities and on seed questions from their field whose true values are or will be known, giving 5th, 50th and 95th percentiles for each. On the seeds, each expert is scored for calibration (do about 90 percent of true values fall inside their 5-95 ranges, with the right split across bins?) and informativeness (how narrow their ranges are). Weights combine both, built so that calibration dominates and honest reporting is rewarded. The weighted combination, called the decision maker, usually outperforms both the best single expert and equal weighting, which becomes diffuse beyond about eight experts. Structure means experts are named, assessments traceable and calculations transparent.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Select experts and agree the purpose, the target quantities and the information each estimate should be conditioned on.
  2. Write unambiguous questions about observable quantities, and 10 or more seed questions from the same domain whose true values will be known.
  3. Elicit each expert individually for 5th, 50th and 95th percentiles (or 5/25/50/75/95) on target and seed questions, after training on the format.
  4. Score each expert on the seeds for calibration (how often and where true values fall in their ranges) and informativeness (narrowness relative to a background range).
  5. Compute performance-based weights from the scores, following Cooke (1991) for the formulas and any weight cut-off.
  6. Form the weighted combination of the experts' distributions, compare it with equal weighting, and report both with the scores.

Sources

  • Uncertainty Analysis with High Dimensional Dependence Modelling (Kurowicka & Cooke, 2006)

Also known as

  • Classical model
  • Cooke's method
  • Cooke's classical model
  • Performance-based expert weighting
  • Structured expert judgement
  • SEJ
  • Structured expert judgment with performance-based weighting (Cooke's classical model)

Categories

  • Strategy & Collaboration
  • Game Theory & Decision Science