Prior elicitation from qualitative research

Description

Converts evidence and expert judgement from qualitative research into explicit probability distributions that serve as priors for subsequent quantitative tests.

Qualitative research often produces strong beliefs about effects, for example that a trust badge will matter for first-time users, but these beliefs are lost when the A/B test starts from zero. Structured prior elicitation captures them as explicit distributions. Following frameworks such as SHELF, experts who know the qualitative evidence give quantiles (for example median, 25th and 75th percentiles) for the quantity of interest, individually and then through facilitated discussion, avoiding anchoring and overconfidence. A distribution, such as a Beta for a rate or a Normal for a log-odds effect, is fitted to the agreed quantiles. Rigour comes from documenting the evidence base, training elicitors in calibration, using the roulette or quantile method, and running sensitivity analyses so that the prior informs but does not dominate.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Data
Mixed
Effort
Days

Steps

  1. Define the quantity precisely, such as the conversion rate of the control or the relative uplift from the trust badge among first-time users.
  2. Compile an evidence dossier from qualitative studies, past experiments and benchmarks, and share it with 3 to 6 elicitors.
  3. Brief elicitors on biases (anchoring, overconfidence) with a short calibration exercise on known quantities.
  4. Elicit individual judgements privately using quantiles (lower plausible, 25th, median, 75th, upper plausible) or the roulette method.
  5. Hold a facilitated discussion to reach a consensus distribution, or pool individual distributions mathematically, and fit a parametric distribution to the result.
  6. Translate into prior strength (for example a Beta prior equivalent to 50 observations), pre-register it, and plan a sensitivity analysis with a weaker prior.

Sources

  • Statistical Rethinking (McElreath)
  • Bayesian Methods for Interaction & Design
  • Behavioral Drift Playbook (Gopinath)
  • The Inquiry Series, Ch 40 — Bayesian Thinking & Sequential Study Design

Also known as

  • Expert prior elicitation
  • Informative priors from qual
  • Structured prior elicitation

Categories

  • Analytics & Behaviour
  • Statistics & Modelling