Description
Randomise users into two identical experiences to check that the experimentation platform yields false positives at the nominal rate and to estimate metric variance.
In an A/A test, users from the same population are randomised into two groups that receive the same unchanged experience. Because there is no true difference, the comparison tests the experimentation system itself: randomisation, logging, metric pipelines and variance estimates. At alpha 0.05, about 5 per cent of metrics or runs should be significant; more frequent rejections signal a biased split, broken logging or wrong variance estimation. Kohavi and colleagues recommend simulating many A/A tests by replaying stored data with fresh randomisation seeds, plotting p-value histograms per metric and testing them for uniformity. Skewed histograms suggest variance errors, a spike near 0.32 a dominant outlier, and point masses rare non-zero values. A/A tests also check sample ratio against the intended split and supply variance for power calculations. Replay cannot catch performance or caching issues, so live A/A tests should continue alongside A/B tests.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Set up two randomised groups with identical experiences, using the same splits (including unequal ones) as planned A/B tests.
- Run for a normal test duration and collect all planned metrics.
- Check sample sizes in each arm against the intended split.
- Test each metric for a difference and record p-values.
- Repeat many times, live or by replaying stored data with new randomisation seeds, and plot p-value histograms per metric.
- Test histograms for uniformity (for example Kolmogorov-Smirnov or Anderson-Darling) and expect about 5 per cent rejections at alpha 0.05.
- Investigate failing metrics, fix variance, capping or instrumentation, and distrust results produced before the fix.
Sources
- Designing with Data (King et al., 2017)
- Designing the User Interface (Shneiderman et al., 2018)
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
- Product Management for UX People (Crumlish, 2022)
Also known as
- Null test
- Null experiment
- A/A testing
- Experiment system validation
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference