Non-Inferiority and Equivalence Trials for Design Changes

Description

Tests whether a new design is not worse than the current one by more than a pre-specified margin, the right question for simplifications, migrations and cost-saving changes.

Many design changes aim not to improve a metric but to avoid harming it: a cheaper infrastructure, a simpler UI, an accessibility fix, removing a feature. Failing to find a significant difference in a standard A/B test does not prove no harm. Non-inferiority trials define a margin (the largest acceptable loss) in advance and test whether the confidence interval's lower bound lies above minus the margin. The margin must be justified clinically, or here, by business and user impact. Sample sizes are planned for this hypothesis. This brings honest inference to the many 'no significant difference, ship it' decisions in product teams.

Attributes

Loss type
Unmigrated
Effort
Weeks

Steps

  1. Specify the primary metric and justify the non-inferiority margin (e.g. no more than 0.5 percentage point loss in conversion).
  2. Compute sample size for the non-inferiority hypothesis at the desired power.
  3. Run the randomised test, using intention-to-treat and per-protocol analyses.
  4. Compute the one-sided confidence interval (e.g. 97.5%) for the difference.
  5. Conclude non-inferiority only if the lower bound exceeds minus the margin in both analyses.
  6. Report the margin rationale and the CI, not just a p-value.

Sources

  • Piaggio et al., Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement (2012), JAMA
  • FDA, Non-Inferiority Clinical Trials to Establish Effectiveness: Guidance for Industry (2016)

Also known as

  • Non-inferiority testing
  • Non-inferiority margin
  • Do-no-harm tests

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference