Description
Test whether an experiment's observed split of units matches its design; a mismatch signals a bug, and results should not be trusted until it is explained.
If an experiment was designed as 50/50 but delivers 50.5/49.5 over millions of users, something has gone wrong: randomisation bugs, ramp or exclusion logic, bot filters in the pipeline, residual effects after restarts, redirects, or triggering on attributes the treatment itself changes. A chi-squared goodness-of-fit test on the observed counts gives a p-value; very low values (for example below 0.001) indicate a bug. Because the mismatch usually means the groups differ in more than the treatment, all metrics become untrustworthy. About 6% of Microsoft experiments showed SRM, and an MSN bot filter that removed treatment's heaviest users reversed a result to +3.3% once fixed. Mature platforms compute the check automatically and hide scorecards that fail it.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Compute the SRM p-value (chi-squared goodness of fit against the designed split) for every experiment and for each triggered population.
- If the p-value is very low (for example below 0.001), hide or flag the scorecard so nobody acts on it.
- Debug systematically: upstream differences, assignment logic, each pipeline stage including bot filtering, start-time differences, segment and daily ratios, and overlaps with other experiments.
- Fix the issue in analysis only if the cause is fully understood; otherwise fix the bug and rerun the experiment.
- Record SRM causes to improve the platform and prevent recurrence.
Sources
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
Also known as
- SRM test
- Sample ratio test
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference