UX Randomised Controlled Trials (Clinical-Grade A/B Testing)

Description

Runs A/B tests with clinical-trial rigour: pre-specified primary outcomes including user harm, power calculations, multiplicity control, pre-registration and CONSORT-style reporting.

The treatise notes A/B testing is structurally equivalent to an RCT and large companies run thousands, but most are not analysed with clinical rigour: sample sizes set by convenience, outcomes restricted to business metrics, multiple testing ignored and pre-registration rare. A UX Epidemiology approach brings clinical trial methodology to digital experimentation: pre-specified primary and secondary outcomes that include user-meaningful harm measures (guardrails), power calculations based on the effect size that would be practically significant, appropriate handling of multiple testing and peeking (sequential designs), and transparent reporting following CONSORT-equivalent standards. Harm outcomes are monitored as in a data safety monitoring process.

Attributes

Loss type
Vestigial
Stage
Post-launch
Effort
Weeks

Steps

  1. Pre-register hypothesis, primary outcome, harm outcomes and analysis plan.
  2. Calculate sample size for the minimum practically significant effect at 80-90% power.
  3. Randomise at the appropriate unit (user, cluster) and check sample ratio mismatch.
  4. Monitor harm outcomes with pre-defined stopping rules; use sequential methods if peeking.
  5. Analyse per plan with multiplicity correction; report intention-to-treat estimates.
  6. Report CONSORT-style: flow, outcomes, harms, deviations, null results.

Sources

  • Lost Methods in UX — A Complete Treatise (Gopinath)
  • Schulz KF, Altman DG & Moher D (2010) CONSORT 2010 statement. BMJ 340:c332
  • Kohavi R, Tang D & Xu Y (2020) Trustworthy Online Controlled Experiments
  • Friedman LM et al. (2015) Fundamentals of Clinical Trials. 5th ed

Also known as

  • UX RCT
  • Clinical-trial-grade experimentation
  • CONSORT for digital experiments

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference