Description
Lets experimenters monitor results repeatedly and stop early for benefit, harm or futility while controlling the false positive rate.
Peeking at a fixed-horizon A/B test and stopping when p < 0.05 inflates false positives dramatically, to 20-30% with frequent looks. Sequential methods fix this. Wald's SPRT compares the likelihood ratio of two hypotheses after each observation against boundaries set by desired error rates, typically requiring far fewer observations on average. Clinical trials use group sequential designs with a few planned interim analyses and alpha-spending functions (O'Brien-Fleming boundaries are conservative early). Mixture SPRT gives always-valid p-values for continuous monitoring. These methods let teams stop harmful tests early and avoid wasted traffic, legitimately.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Pre-specify hypotheses, minimum detectable effect, alpha, power and the monitoring plan (continuous or K interim looks).
- Choose a method: SPRT or mSPRT for continuous monitoring; O'Brien-Fleming or Pocock boundaries via alpha spending for planned looks.
- Compute boundaries and expected sample size; register the plan.
- Monitor at planned points, stopping when a boundary is crossed for efficacy, harm or futility.
- Report the stopping time, adjusted estimates (correcting for bias from early stopping) and confidence intervals.
- Never switch to naive tests mid-experiment.
Sources
- Wald, Sequential Analysis (1947)
- Jennison & Turnbull, Group Sequential Methods with Applications to Clinical Trials (2000)
- Johari, Koomen, Pekelis & Walsh, Peeking at A/B tests (2017), KDD
- Designing Experiments and Analyzing Data (Scott Maxwell & Harold Delaney)
- Methods Atlas (Vivek Gopinath)
- Forward-Thinking Playbook (Vivek Gopinath)
- Group Sequential Methods with Applications to Clinical Trials (Christopher Jennison & Bruce Turnbull)
- Bandit Algorithms (Tor Lattimore & Csaba Szepesvári)
- Designing Experiments & Analyzing Data (Maxwell/Delaney)
- Methods Atlas (Gopinath)
- Forward-Thinking Playbook (Gopinath)
- The Inquiry Series, Ch 47 — Experimental Design — From A/B to Quasi-Experiments
Also known as
- SPRT
- Sequential probability ratio test
- Group sequential design
- Alpha spending
- Always-valid p-values
- Multi-armed bandits
- Sequential testing
- Response-adaptive randomisation
- Adaptive trials
- Adaptive experimental designs
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference