Description
Runs A/B tests with clinical-trial rigour: pre-specified primary outcomes including user harm, power calculations, multiplicity control, pre-registration and CONSORT-style reporting.
The treatise notes A/B testing is structurally equivalent to an RCT and large companies run thousands, but most are not analysed with clinical rigour: sample sizes set by convenience, outcomes restricted to business metrics, multiple testing ignored and pre-registration rare. A UX Epidemiology approach brings clinical trial methodology to digital experimentation: pre-specified primary and secondary outcomes that include user-meaningful harm measures (guardrails), power calculations based on the effect size that would be practically significant, appropriate handling of multiple testing and peeking (sequential designs), and transparent reporting following CONSORT-equivalent standards. Harm outcomes are monitored as in a data safety monitoring process.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Vestigial
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Pre-register hypothesis, primary outcome, harm outcomes and analysis plan.
- Calculate sample size for the minimum practically significant effect at 80-90% power.
- Randomise at the appropriate unit (user, cluster) and check sample ratio mismatch.
- Monitor harm outcomes with pre-defined stopping rules; use sequential methods if peeking.
- Analyse per plan with multiplicity correction; report intention-to-treat estimates.
- Report CONSORT-style: flow, outcomes, harms, deviations, null results.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Schulz KF, Altman DG & Moher D (2010) CONSORT 2010 statement. BMJ 340:c332
- Kohavi R, Tang D & Xu Y (2020) Trustworthy Online Controlled Experiments
- Friedman LM et al. (2015) Fundamentals of Clinical Trials. 5th ed
Also known as
- UX RCT
- Clinical-trial-grade experimentation
- CONSORT for digital experiments
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference