Equivalence testing (TOST)

Description

Tests whether an effect is small enough to be considered practically equivalent to zero, allowing teams to conclude 'no meaningful difference' rather than merely 'not significant'.

A non-significant result does not show that two designs are equivalent; it may reflect low power. Equivalence testing reverses the logic: you set equivalence bounds, the smallest effect that matters (for example ±2 percentage points of conversion or ±0.3 SEQ points), and run two one-sided tests to check whether the effect is significantly above the lower bound and significantly below the upper bound. If both are significant, you conclude equivalence. Equivalently, the 90% confidence interval lies entirely within the bounds. Non-inferiority tests use only one bound, asking whether a new design is not meaningfully worse. In UX this supports decisions like simplifying a flow, migrating platforms or adding a guardrail without harming a metric. Rigour comes from pre-specifying bounds based on practical relevance and powering the test for equivalence.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Set equivalence bounds based on the smallest effect of interest, justified by business or user impact, for example ±0.25 standard deviations or ±1.5 percentage points.
  2. Plan the sample for equivalence power (often larger than for a difference test), using tools such as the TOSTER package or G*Power-like calculators.
  3. Collect data and run two one-sided tests at alpha 0.05: H01 effect ≤ lower bound and H02 effect ≥ upper bound.
  4. Conclude equivalence if both nulls are rejected, which corresponds to the 90% confidence interval lying within the bounds.
  5. Report the TOST p-values, the 90% and 95% intervals, and the ordinary difference test so readers see both perspectives.
  6. For one-sided questions such as 'the new flow is not worse', run a non-inferiority test with a single margin.

Sources

  • Statistics for Dummies series (Rumsey)
  • Designing Experiments & Analyzing Data (Maxwell & Delaney)
  • scipy.stats
  • pingouin
  • statsmodels
  • The Inquiry Series, Ch 41 — Frequentist Inference Done Right
  • mord
  • PyMC
  • bambi
  • matplotlib
  • seaborn
  • The Inquiry Series, C2 — Quantitative UX Analysis in Python — Descriptive to Inferential

Also known as

  • Two one-sided tests
  • TOST procedure
  • Non-inferiority testing
  • Equivalence bounds testing

Categories

  • Analytics & Behaviour
  • Statistics & Modelling