Description
Uses each user's pre-experiment metric as a covariate to remove predictable variance from A/B test outcomes, shrinking confidence intervals substantially.
CUPED adjusts each user's outcome Y by subtracting theta times (X minus mean X), where X is a pre-period covariate (often the same metric before the experiment) and theta is cov(X,Y)/var(X). Because X is measured before assignment, it is independent of treatment, so the estimate remains unbiased while variance falls by a factor of 1 - rho squared. With typical correlations of 0.5-0.7 this cuts required sample sizes by 25-50%. It is a control-variate technique well known in Monte Carlo and survey sampling. UX-led experiments, which are often underpowered, benefit greatly, yet many teams compare raw means.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Choose a pre-period covariate strongly correlated with the outcome (e.g. same metric over the prior 2-4 weeks).
- Ensure the covariate is measured strictly before randomisation; handle new users without history (impute or use other covariates).
- Compute theta from pooled data across both arms.
- Compute adjusted outcomes and estimate the treatment effect and its standard error.
- Report variance reduction achieved and compare with the unadjusted estimate as a sanity check.
- Use the expected variance reduction in power calculations for future tests.
Sources
- Deng, Xu, Kohavi & Walker, Improving the sensitivity of online controlled experiments by utilizing pre-experiment data (2013), WSDM
- Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)
Also known as
- CUPED
- Controlled experiments using pre-experiment data
- Regression adjustment in A/B tests
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference