Description
Tests whether an effect is small enough to be considered practically equivalent to zero, allowing teams to conclude 'no meaningful difference' rather than merely 'not significant'.
A non-significant result does not show that two designs are equivalent; it may reflect low power. Equivalence testing reverses the logic: you set equivalence bounds, the smallest effect that matters (for example ±2 percentage points of conversion or ±0.3 SEQ points), and run two one-sided tests to check whether the effect is significantly above the lower bound and significantly below the upper bound. If both are significant, you conclude equivalence. Equivalently, the 90% confidence interval lies entirely within the bounds. Non-inferiority tests use only one bound, asking whether a new design is not meaningfully worse. In UX this supports decisions like simplifying a flow, migrating platforms or adding a guardrail without harming a metric. Rigour comes from pre-specifying bounds based on practical relevance and powering the test for equivalence.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Latent
- Goal
- Compare designs
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Set equivalence bounds based on the smallest effect of interest, justified by business or user impact, for example ±0.25 standard deviations or ±1.5 percentage points.
- Plan the sample for equivalence power (often larger than for a difference test), using tools such as the TOSTER package or G*Power-like calculators.
- Collect data and run two one-sided tests at alpha 0.05: H01 effect ≤ lower bound and H02 effect ≥ upper bound.
- Conclude equivalence if both nulls are rejected, which corresponds to the 90% confidence interval lying within the bounds.
- Report the TOST p-values, the 90% and 95% intervals, and the ordinary difference test so readers see both perspectives.
- For one-sided questions such as 'the new flow is not worse', run a non-inferiority test with a single margin.
Sources
- Statistics for Dummies series (Rumsey)
- Designing Experiments & Analyzing Data (Maxwell & Delaney)
- scipy.stats
- pingouin
- statsmodels
- The Inquiry Series, Ch 41 — Frequentist Inference Done Right
- mord
- PyMC
- bambi
- matplotlib
- seaborn
- The Inquiry Series, C2 — Quantitative UX Analysis in Python — Descriptive to Inferential
Also known as
- Two one-sided tests
- TOST procedure
- Non-inferiority testing
- Equivalence bounds testing
Categories
- Analytics & Behaviour
- Statistics & Modelling