Multiple comparisons correction

Description

Controls the inflated chance of false positives when many metrics, segments or variants are tested, using family-wise or false discovery rate procedures.

With an alpha of 0.05, testing 20 independent null comparisons gives about a 64% chance of at least one false positive. UX analyses routinely test many metrics, segments, variants and survey items, then report the significant ones. Corrections control this inflation. Family-wise error rate (FWER) methods, Bonferroni (alpha divided by number of tests) and the more powerful Holm step-down procedure, limit the probability of any false positive and suit confirmatory decisions. False discovery rate (FDR) methods, notably Benjamini-Hochberg, limit the expected proportion of false positives among discoveries and suit exploratory screening of many metrics. Rigour also includes reducing the number of tests through a pre-registered primary metric and treating secondary findings as exploratory.

Attributes

Loss type
Vestigial
Effort
Hours

Steps

  1. List all planned comparisons: metrics, segments, variants and time windows; designate one or a few primary outcomes in advance.
  2. Decide the error rate to control: FWER for confirmatory decisions (use Holm), or FDR for exploratory screening (use Benjamini-Hochberg at q = 0.05 or 0.10).
  3. For Holm, sort p-values ascending and compare the k-th smallest with alpha/(m − k + 1), stopping at the first non-significant result.
  4. For Benjamini-Hochberg, sort p-values and find the largest k where p(k) ≤ (k/m)q; reject hypotheses 1 to k.
  5. Report both raw and adjusted p-values, along with effect sizes and intervals.
  6. Label unplanned or segment-level findings as exploratory and confirm important ones in a follow-up test.

Sources

  • Statistics for Dummies series (Rumsey)
  • Designing Experiments & Analyzing Data (Maxwell & Delaney)
  • The Inquiry Series, Ch 41 — Frequentist Inference Done Right
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • Multiple testing correction
  • Family-wise error control
  • False discovery rate control
  • Bonferroni and Holm corrections

Categories

  • Analytics & Behaviour
  • Statistics & Modelling