Description
Compares success rates when the same participants tried both designs.
In within-subjects studies each person attempts the task on both designs, so the two completion rates are not independent and the two-proportion test is wrong. McNemar's test looks only at the discordant pairs: people who succeeded on A but failed on B, and the reverse. If the designs were equally good these should be roughly equal. The exact version uses a binomial test on the discordant counts and works with small samples. Sauro and Lewis pair it with an adjusted-Wald interval for the difference in matched proportions, so the size of the improvement can be reported too.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Siloed
- Goal
- Compare designs
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Build the 2×2 table of outcomes per participant: both succeed, A only, B only, both fail.
- Take the discordant counts b (A only) and c (B only).
- Run an exact binomial test of b against b + c with p = 0.5 (or the mid-p variant for less conservatism).
- Compute an adjusted interval for the difference in matched proportions.
- Report the discordant counts, p-value and interval.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Benchmarking the User Experience (Sauro)
Also known as
- McNemar test
- Dependent proportions test
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference