McNemar Exact Test for Paired Proportions

Description

Compares success rates when the same participants tried both designs.

In within-subjects studies each person attempts the task on both designs, so the two completion rates are not independent and the two-proportion test is wrong. McNemar's test looks only at the discordant pairs: people who succeeded on A but failed on B, and the reverse. If the designs were equally good these should be roughly equal. The exact version uses a binomial test on the discordant counts and works with small samples. Sauro and Lewis pair it with an adjusted-Wald interval for the difference in matched proportions, so the size of the improvement can be reported too.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Build the 2×2 table of outcomes per participant: both succeed, A only, B only, both fail.
  2. Take the discordant counts b (A only) and c (B only).
  3. Run an exact binomial test of b against b + c with p = 0.5 (or the mid-p variant for less conservatism).
  4. Compute an adjusted interval for the difference in matched proportions.
  5. Report the discordant counts, p-value and interval.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Benchmarking the User Experience (Sauro)

Also known as

  • McNemar test
  • Dependent proportions test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference