Bayesian Sequential Design

Description

Collects data in batches and stops when a pre-specified posterior or Bayes factor criterion is met, using fewer participants than fixed-N designs while controlling decision error.

Fixed-sample studies commit to a sample size in advance, often wasting participants when effects are clear or stopping too early when they are not. Bayesian sequential designs update the posterior after each batch and stop when a pre-registered criterion is met: for example, the probability that variant B beats A exceeds 95 percent, a Bayes factor exceeds 10 for either hypothesis, or the credible interval is narrower than a target precision. Because Bayesian inference conditions on observed data, monitoring does not invalidate it in the way repeated significance testing inflates frequentist error, though operating characteristics should still be checked by simulation. Priors can be informed by earlier qualitative or quantitative work. Rigour comes from pre-specifying priors, stopping rules and maximum sample size.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Weeks

Steps

  1. Define the decision, the outcome metric and the model (for example Beta-Binomial for task success or conversion).
  2. Specify priors, using weakly informative priors or ones informed by previous studies, and document them.
  3. Pre-register the stopping rule (for example posterior probability above 0.95, Bayes factor above 10 or below 1/10, or a target interval width) and a maximum sample size.
  4. Simulate the design to check expected sample size and error rates under plausible effect sizes.
  5. Collect data in batches, updating the posterior after each batch and stopping as soon as the criterion or the maximum is reached.
  6. Report the posterior, the stopping point, prior sensitivity and the full sequence of interim results.

Sources

  • Statistical Rethinking (McElreath)
  • Bayesian Methods for Interaction & Design
  • Behavioral Drift Playbook (Gopinath)
  • Bayesian Data Analysis (Gelman et al.)
  • The Inquiry Series, Vol III, Ch 40
  • The Inquiry Series, Ch 40 — Bayesian Thinking & Sequential Study Design

Also known as

  • Sequential Bayesian stopping rules
  • Bayesian adaptive design
  • Sequential Bayes factor design

Categories

  • Analytics & Behaviour
  • Statistics & Modelling