Prior and posterior predictive checks

Description

Simulates data from a model's priors before fitting, and from its posterior after fitting, to see whether its assumptions imply plausible outcomes and where it misdescribes the data.

Prior predictive simulation draws parameter values from the priors alone and pushes them through the model to produce simulated outcomes, showing in the units of the outcome what harmless-looking priors imply. McElreath's example: an intercept prior of Normal(178, 100) gave 4% of simulated people negative height. Flat priors on logit scales pile probability at 0 and 1. Priors are then tightened using pre-data knowledge only. Posterior predictive checks run after fitting: for every posterior draw a data set of the same size is simulated, carrying full parameter uncertainty, and the simulations are compared with the real data. First compare on the summary the model uses, as a software check; then on features it does not use, chosen with domain knowledge, such as runs, zero counts or per-participant patterns. The aim is to find where the model fails and whether that matters, not to test truth.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Write the full model with a prior for every parameter, using knowledge of ranges, signs and theory but never the sample itself.
  2. Draw 100 to 10,000 values from the priors, simulate outcomes or regression lines, and plot them against known limits (no negative task times, conversion between 0 and 1).
  3. Reshape priors that allow absurd outcomes (for example standardise variables and use Normal(0, 1) slopes, or log-normal priors for positive effects) and repeat; for link functions check implied probabilities.
  4. Fit the model and, for each posterior draw, simulate a data set of the same size as the observed one.
  5. Compare simulated and observed data on the statistic the model fits, then on features it does not use, such as longest run, share of zeros or per-participant spread.
  6. Judge whether any misfit matters for the decision, revise the model where it does, and report the checks and prior sensitivity.

Sources

  • Statistical Rethinking (McElreath, 2020)
  • Applied Bayesian Modelling (Congdon, 2003)
  • The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)
  • Bayesian Analysis for the Social Sciences (Jackman, 2009)
  • Structural Equation Modeling: A Bayesian Approach (Lee, 2007)
  • Bayesian Statistical Modelling (Congdon, 2006)

Also known as

  • Posterior predictive checks
  • PPC
  • Posterior predictive simulation
  • Retrodictive check
  • Prior predictive simulation
  • Prior predictive check
  • Prior simulation

Categories

  • Analytics & Behaviour
  • Statistics & Modelling