CUPED Variance Reduction

Description

Uses each user's pre-experiment metric as a covariate to remove predictable variance from A/B test outcomes, shrinking confidence intervals substantially.

CUPED adjusts each user's outcome Y by subtracting theta times (X minus mean X), where X is a pre-period covariate (often the same metric before the experiment) and theta is cov(X,Y)/var(X). Because X is measured before assignment, it is independent of treatment, so the estimate remains unbiased while variance falls by a factor of 1 - rho squared. With typical correlations of 0.5-0.7 this cuts required sample sizes by 25-50%. It is a control-variate technique well known in Monte Carlo and survey sampling. UX-led experiments, which are often underpowered, benefit greatly, yet many teams compare raw means.

Attributes

Loss type
Latent
Stage
Post-launch
Effort
Hours

Steps

  1. Choose a pre-period covariate strongly correlated with the outcome (e.g. same metric over the prior 2-4 weeks).
  2. Ensure the covariate is measured strictly before randomisation; handle new users without history (impute or use other covariates).
  3. Compute theta from pooled data across both arms.
  4. Compute adjusted outcomes and estimate the treatment effect and its standard error.
  5. Report variance reduction achieved and compare with the unadjusted estimate as a sanity check.
  6. Use the expected variance reduction in power calculations for future tests.

Sources

  • Deng, Xu, Kohavi & Walker, Improving the sensitivity of online controlled experiments by utilizing pre-experiment data (2013), WSDM
  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)

Also known as

  • CUPED
  • Controlled experiments using pre-experiment data
  • Regression adjustment in A/B tests

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference