Permutation Test (Resampling Significance Test)

Description

Tests whether two groups really differ by shuffling the group labels thousands of times to see how often chance alone gives a gap as large.

Instead of relying on formulas and distributional assumptions, a permutation test asks directly: if the design made no difference, how often would randomly reassigning participants to groups produce a difference as large as the one observed? Pool all observations, shuffle and split them into groups of the original sizes many times, and compute the difference each time. The proportion of shuffled differences at least as extreme as the real one is the p-value. Bruce and Bruce present it as the data scientist's default because it works for any statistic, including medians, ratios and session stickiness, and is easy to explain to stakeholders.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Effort
Hours

Steps

  1. Compute the observed statistic (difference in means, medians, rates, anything).
  2. Pool the data from both groups.
  3. Repeatedly shuffle and split the pool into groups of the original sizes, recomputing the statistic (1,000 to 10,000 times).
  4. Calculate the proportion of permuted statistics as or more extreme than the observed one.
  5. Report this p-value with a histogram of the permutation distribution.

Sources

  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • Randomisation test
  • Exact permutation test

Categories

  • Analytics & Behaviour
  • Statistics & Modelling