Description
Tests whether two groups really differ by shuffling the group labels thousands of times to see how often chance alone gives a gap as large.
Instead of relying on formulas and distributional assumptions, a permutation test asks directly: if the design made no difference, how often would randomly reassigning participants to groups produce a difference as large as the one observed? Pool all observations, shuffle and split them into groups of the original sizes many times, and compute the difference each time. The proportion of shuffled differences at least as extreme as the real one is the p-value. Bruce and Bruce present it as the data scientist's default because it works for any statistic, including medians, ratios and session stickiness, and is easy to explain to stakeholders.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Goal
- Compare designs
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Compute the observed statistic (difference in means, medians, rates, anything).
- Pool the data from both groups.
- Repeatedly shuffle and split the pool into groups of the original sizes, recomputing the statistic (1,000 to 10,000 times).
- Calculate the proportion of permuted statistics as or more extreme than the observed one.
- Report this p-value with a histogram of the permutation distribution.
Sources
- Practical Statistics for Data Scientists (Bruce & Bruce)
Also known as
- Randomisation test
- Exact permutation test
Categories
- Analytics & Behaviour
- Statistics & Modelling