Description
A pre-agreed decision procedure that maps an experiment's result, confidence interval and practical-significance threshold to ship, iterate, abandon or rerun, before anyone sees the data.
Most teams run A/B tests but decide ad hoc once results arrive. This method fixes the decision rule first. The team chooses the decision type (superiority, bias to ship or non-inferiority, agnostic, measurement, or lean-startup exploration), the key metric and the smallest change worth acting on, and translates verbal certainty into numbers (for example 'almost certainly' as 95%, 'quite likely' as 80%). After the planned sample, the confidence interval is compared with both zero and the practical boundary, giving six cases from clear launch to rerun with more power. Results are then triaged as expected, unexpected or flat, each with its own questions: user disruption and maintenance cost for winners, execution faults and secondary metrics for losers, and power, duration and cell differentiation for flat results. Rigour comes from pre-commitment and from separating statistical from practical significance.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Vestigial
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Before launch, agree the decision type with stakeholders: superiority, bias to ship (non-inferiority), agnostic, measurement or lean-startup exploration.
- Choose one key metric and fix the practical significance boundary, the smallest change worth acting on (for example 1% relative), plus the certainty levels in agreed phrases mapped to numbers (95% 'almost certainly', 80% 'quite likely').
- Run a power analysis for that rule, start the test, validate the data (sample ratio, logging) and wait for the planned sample without peeking.
- Compute the confidence interval of the delta and classify it: not significant with a narrow interval (iterate or abandon); statistically and practically significant (launch); significant but below the practical boundary (launch only if cheap); or an interval spanning the practical boundary (rerun with more power).
- For an expected result, weigh user disruption, retraining, support and maintenance costs and any shortcuts taken in the test before shipping.
- For an unexpected (negative) result, question assumptions about use and motivation, read the secondary metrics, add qualitative research and check for execution faults; rework the hypothesis rather than polishing the failed design.
- For a flat result, check sample, duration, business cycles, external events, how different the cells really were and upstream flow effects; retest only with other supporting evidence.
- Record the decision in the agreed phrases: roll out to 100%, widen exposure, or keep the existing experience, and plan communication for global changes.
Sources
- Designing with Data (King et al., 2017)
- Quant UX Con 2022 Proceedings (Chapman et al., 2022)
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
Related talks
Also known as
- Post-test decision tree
- Ship, iterate or abandon decision
- Decision-type framing for A/B tests
- Pre-agreed A/B decision rules
- Launch decision by statistical versus practical significance
- Six-case launch decision matrix
- A/B result triage
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference