Sample ratio mismatch (SRM) check

Description

Test whether an experiment's observed split of units matches its design; a mismatch signals a bug, and results should not be trusted until it is explained.

If an experiment was designed as 50/50 but delivers 50.5/49.5 over millions of users, something has gone wrong: randomisation bugs, ramp or exclusion logic, bot filters in the pipeline, residual effects after restarts, redirects, or triggering on attributes the treatment itself changes. A chi-squared goodness-of-fit test on the observed counts gives a p-value; very low values (for example below 0.001) indicate a bug. Because the mismatch usually means the groups differ in more than the treatment, all metrics become untrustworthy. About 6% of Microsoft experiments showed SRM, and an MSN bot filter that removed treatment's heaviest users reversed a result to +3.3% once fixed. Mature platforms compute the check automatically and hide scorecards that fail it.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Compute the SRM p-value (chi-squared goodness of fit against the designed split) for every experiment and for each triggered population.
  2. If the p-value is very low (for example below 0.001), hide or flag the scorecard so nobody acts on it.
  3. Debug systematically: upstream differences, assignment logic, each pipeline stage including bot filtering, start-time differences, segment and daily ratios, and overlaps with other experiments.
  4. Fix the issue in analysis only if the cause is fully understood; otherwise fix the bug and rerun the experiment.
  5. Record SRM causes to improve the platform and prevent recurrence.

Sources

  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)

Also known as

  • SRM test
  • Sample ratio test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference