N-1 Two-Proportion Test

Description

Compares two independent success or conversion rates accurately, even when samples are small.

Comparing completion or conversion rates between two designs, tested on different people, is the most common quantitative UX comparison. The classic chi-square test is inaccurate when expected cell counts are small, and Fisher's exact test is overly conservative, so real differences get missed. Campbell's simulation work showed that multiplying the chi-square statistic by (N−1)/N gives accurate Type I error rates across almost all sample sizes, as long as every expected count is at least one. The equivalent z-test version also yields a confidence interval around the difference, which tells stakeholders how big the improvement might be, not only whether it is real.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Record successes and totals for each independent group.
  2. Check that every expected cell count is at least 1; if not, use Fisher's exact test.
  3. Compute the pooled proportion and z = (p1 − p2) / sqrt(P(1−P)(1/n1 + 1/n2)), then multiply by sqrt((N−1)/N).
  4. Convert to a two-tailed p-value.
  5. Compute an adjusted-Wald interval around the difference and report both.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Benchmarking the User Experience (Sauro)

Also known as

  • N-1 chi-square test
  • Campbell N-1 test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference