Delta-method variance for ratio metrics

Description

Corrects the variance of ratio metrics such as click-through rate, and of percent changes, when the analysis unit is finer than the randomisation unit.

When an experiment randomises users but the metric is computed per page view, session or click, observations from the same user are correlated and the naive binomial variance is too small. Confidence intervals are then too narrow and false positives rise well above the nominal rate. The fix is to express the metric as a ratio of two per-user averages (for example clicks per user over page views per user) and apply the delta method, a first-order Taylor approximation that combines the variances and covariance of numerator and denominator. Percent-change results need the same treatment because the Control mean in the denominator is itself random. Where no ratio form exists, such as percentiles, the bootstrap or density estimation is used. The rigour lies in matching the variance estimator to the randomisation unit and confirming the correction with A/A tests whose p-values should be uniform.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. List every metric whose analysis unit (page, session, click) differs from the randomisation unit (usually the user or device).
  2. Rewrite each such metric as a ratio of two per-user means, aggregating raw events to one row per randomisation unit.
  3. Compute the delta-method variance from the variances of numerator and denominator and their covariance, and build the confidence interval from it.
  4. For percent-change estimates, apply the delta method to the ratio of Treatment to Control means rather than treating the Control mean as fixed.
  5. Use the bootstrap, resampling whole users, for statistics with no ratio form such as medians or 90th percentiles.
  6. Run A/A tests and check that p-values are uniform and that about 5% fall below 0.05; a higher rate means the variance is still understated.

Sources

  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • Advanced Calculus with Applications in Statistics (Khuri, 2003)
  • Mathematical Statistics with Resampling and R (Chihara & Hesterberg, 2011)
  • Statistical Inference (Casella & Berger, 2001)

Also known as

  • Delta method
  • Ratio metric variance
  • Delta-method variance for ratio and percent-change metrics

Categories

  • Analytics & Behaviour
  • Statistics & Modelling