Description
Keeps an informative prior's mean fixed while scaling its precision, plots how the posterior and decision change, and reports the prior weight in equivalent respondents needed to reverse the conclusion.
An informative prior built from past data (earlier releases, benchmarks, previous studies) can dominate a small new study. Rather than argue about whether to use it, this analysis sweeps its weight. A conjugate prior such as Beta(a, b) is rewritten as Beta(a/lambda, b/lambda): the prior mean is unchanged while its effective sample size, (a + b)/lambda, shrinks as lambda grows. Because the posterior mean is a precision-weighted average of the prior mean and the data estimate, sweeping lambda from very small (prior dominates) to very large (data dominate) traces how inference moves between them. The decision quantity, for example the probability that a conversion rate exceeds a target, is plotted against lambda, and the point where it crosses the decision threshold is translated into an equivalent number of observations. In Jackman's example only a prior worth about 2,700 respondents would have reversed the poll's call, a concrete number decision-makers can judge.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Latent
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Build an informative prior from relevant past data and express it as a conjugate density with a clear effective sample size.
- Introduce a precision multiplier that keeps the prior mean fixed while changing its effective sample size.
- Combine the prior with the current data across a wide grid of multiplier values (for example on a log scale from 0.001 to 1000).
- Plot the posterior mean, 95% interval and the posterior probability of the decision-relevant event against the multiplier.
- Find where the decision probability crosses the decision threshold and convert that multiplier into an equivalent number of observations.
- Judge whether that much prior weight is plausible given how exchangeable past and present data are, and report the sweep alongside the main result; vary the prior mean separately if it is contested.
Sources
- Bayesian Analysis for the Social Sciences (Jackman, 2009)
- Applied Bayesian Modelling (Congdon, 2003)
- Bayesian Networks: An Introduction (Koski & Noble, 2009)
- Bayes Linear Statistics: Theory and Methods (Goldstein & Wooff, 2007)
Also known as
- Prior discounting analysis
- Power-prior style sensitivity sweep
- Equivalent prior sample size tipping point
Categories
- Analytics & Behaviour
- Statistics & Modelling