Description
Controls the inflated chance of false positives when many metrics, segments or variants are tested, using family-wise or false discovery rate procedures.
With an alpha of 0.05, testing 20 independent null comparisons gives about a 64% chance of at least one false positive. UX analyses routinely test many metrics, segments, variants and survey items, then report the significant ones. Corrections control this inflation. Family-wise error rate (FWER) methods, Bonferroni (alpha divided by number of tests) and the more powerful Holm step-down procedure, limit the probability of any false positive and suit confirmatory decisions. False discovery rate (FDR) methods, notably Benjamini-Hochberg, limit the expected proportion of false positives among discoveries and suit exploratory screening of many metrics. Rigour also includes reducing the number of tests through a pre-registered primary metric and treating secondary findings as exploratory.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Vestigial
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- List all planned comparisons: metrics, segments, variants and time windows; designate one or a few primary outcomes in advance.
- Decide the error rate to control: FWER for confirmatory decisions (use Holm), or FDR for exploratory screening (use Benjamini-Hochberg at q = 0.05 or 0.10).
- For Holm, sort p-values ascending and compare the k-th smallest with alpha/(m − k + 1), stopping at the first non-significant result.
- For Benjamini-Hochberg, sort p-values and find the largest k where p(k) ≤ (k/m)q; reject hypotheses 1 to k.
- Report both raw and adjusted p-values, along with effect sizes and intervals.
- Label unplanned or segment-level findings as exploratory and confirm important ones in a follow-up test.
Sources
- Statistics for Dummies series (Rumsey)
- Designing Experiments & Analyzing Data (Maxwell & Delaney)
- The Inquiry Series, Ch 41 — Frequentist Inference Done Right
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Quantifying the User Experience (Sauro & Lewis)
Also known as
- Multiple testing correction
- Family-wise error control
- False discovery rate control
- Bonferroni and Holm corrections
Categories
- Analytics & Behaviour
- Statistics & Modelling