Description
Randomly assigns users or participants to design conditions so differences in outcomes can be attributed to the design, with power, metrics and analysis set in advance.
An RCT is the reference design for causal claims: randomisation balances known and unknown confounders in expectation, so outcome differences estimate the effect of the intervention. A/B testing is its online form, but UX practice often strips out the discipline that makes it rigorous: pre-specified primary metric, sample size from a power calculation, a fixed stopping rule, checks for sample ratio mismatch, guardrail metrics and a pre-registered analysis plan. Fisher's principles of randomisation, replication and blocking still apply, along with clinical-trial ideas such as intention-to-treat analysis and CONSORT-style reporting. Lab RCTs comparing prototypes, with counterbalancing and within-subject designs, are the small-sample sibling of online tests.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Vestigial
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Write a hypothesis, one primary metric, guardrail metrics and the minimum detectable effect that would change the decision.
- Compute sample size for alpha 0.05 and power 0.8 (or 0.9 for high stakes), accounting for variance and the unit of randomisation.
- Choose the randomisation unit (user, account, session, region) to avoid interference and contamination between arms.
- Pre-register the analysis plan, including stopping rule, segments and handling of outliers.
- Run an A/A test or check sample ratio mismatch (chi-squared p below 0.001 signals a fault) before reading results.
- Run for the full planned duration and at least one full weekly cycle; avoid peeking unless using a sequential method.
- Analyse by intention-to-treat, report effect size with confidence intervals and guardrail results, and document the decision.
Sources
- Designing Experiments and Analyzing Data (Scott Maxwell & Harold Delaney)
- Methods Atlas (Vivek Gopinath)
- Forward-Thinking Playbook (Vivek Gopinath)
- Trustworthy Online Controlled Experiments (Ron Kohavi, Diane Tang & Ya Xu)
- The Design of Experiments (Ronald Fisher)
- Designing Experiments & Analyzing Data (Maxwell/Delaney)
- Methods Atlas (Gopinath)
- Forward-Thinking Playbook (Gopinath)
- The Inquiry Series, Ch 47 — Experimental Design — From A/B to Quasi-Experiments
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
- Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)
Also known as
- Randomised controlled trial
- A/B test design
- Controlled experiment
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference