Description
Applies the correct meaning of a p-value, the probability of data at least this extreme if the null model were true, and avoids its common misreadings in UX decisions.
A p-value is the probability, assuming the null hypothesis and all model assumptions are true, of observing data at least as extreme as those observed. It is not the probability that the null is true, not the probability that the result is due to chance, and not a measure of effect size or importance. The ASA's 2016 statement set out six principles, including that scientific conclusions should not rest only on whether p passes 0.05. In UX, misinterpretation leads to shipping trivial effects with large samples, dismissing important effects with small samples, and treating p = 0.06 and p = 0.04 as opposites. Correct interpretation pairs p-values with effect sizes and confidence intervals, pre-specified alpha and power, and attention to multiple testing and analytic flexibility.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Vestigial
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- State the null and alternative hypotheses and the alpha level (commonly 0.05) before seeing data, along with the planned test.
- Check the test's assumptions (independence, distribution, equal variances or appropriate corrections) and choose an alternative test if violated.
- Report the exact p-value (for example p = 0.032, not p < 0.05) together with the effect size and its 95% confidence interval.
- Interpret using correct wording: 'If there were no true difference, data this extreme would occur about 3% of the time', not 'there is a 97% chance the variant is better'.
- Judge practical significance against a pre-defined smallest effect of interest, separately from statistical significance.
- For non-significant results, report the interval and power, and do not claim 'no effect' unless an equivalence test supports it.
Sources
- Statistics for Dummies series (Rumsey)
- Designing Experiments & Analyzing Data (Maxwell & Delaney)
- The Inquiry Series, Ch 41 — Frequentist Inference Done Right
Also known as
- P-value literacy
- Correct NHST interpretation
- Significance testing done right
Categories
- Analytics & Behaviour
- Statistics & Modelling