A/A test

Description

Randomise users into two identical experiences to check that the experimentation platform yields false positives at the nominal rate and to estimate metric variance.

In an A/A test, users from the same population are randomised into two groups that receive the same unchanged experience. Because there is no true difference, the comparison tests the experimentation system itself: randomisation, logging, metric pipelines and variance estimates. At alpha 0.05, about 5 per cent of metrics or runs should be significant; more frequent rejections signal a biased split, broken logging or wrong variance estimation. Kohavi and colleagues recommend simulating many A/A tests by replaying stored data with fresh randomisation seeds, plotting p-value histograms per metric and testing them for uniformity. Skewed histograms suggest variance errors, a spike near 0.32 a dominant outlier, and point masses rare non-zero values. A/A tests also check sample ratio against the intended split and supply variance for power calculations. Replay cannot catch performance or caching issues, so live A/A tests should continue alongside A/B tests.

Attributes

Loss type
Active
Effort
Days

Steps

  1. Set up two randomised groups with identical experiences, using the same splits (including unequal ones) as planned A/B tests.
  2. Run for a normal test duration and collect all planned metrics.
  3. Check sample sizes in each arm against the intended split.
  4. Test each metric for a difference and record p-values.
  5. Repeat many times, live or by replaying stored data with new randomisation seeds, and plot p-value histograms per metric.
  6. Test histograms for uniformity (for example Kolmogorov-Smirnov or Anderson-Darling) and expect about 5 per cent rejections at alpha 0.05.
  7. Investigate failing metrics, fix variance, capping or instrumentation, and distrust results produced before the fix.

Sources

  • Designing with Data (King et al., 2017)
  • Designing the User Interface (Shneiderman et al., 2018)
  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • Product Management for UX People (Crumlish, 2022)

Also known as

  • Null test
  • Null experiment
  • A/A testing
  • Experiment system validation

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference