Sample Size Planning for Summative UX Studies

Description

Works out how many participants a benchmark or comparison needs to hit a target precision or detect a meaningful difference.

Summative studies measure rather than discover, so sample size follows from statistics, not the five-user rule. There are three situations. For a stand-alone benchmark, n depends on the margin of error you can accept around a completion rate or mean. For comparisons, n depends on the smallest difference worth detecting, the variability, the confidence level and the power, and within-subjects designs need far fewer people than between-subjects ones. For comparisons against an industry benchmark, n depends on how close to the benchmark you expect to be. Sauro and Lewis give iterative t-based formulas that work for small samples, and practical tables, such as roughly 93 people for a ±10 point margin on a completion rate.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Classify the study: stand-alone estimate, comparison (within or between), or test against a benchmark.
  2. Choose confidence level and, for comparisons, power (80% is typical).
  3. Get a variability estimate from past studies (SUS SD is about 17.7; for proportions use 0.5 if unknown).
  4. Apply the formula or table, iterating the t value until n stabilises.
  5. Inflate for expected drop-outs and data cleaning losses (often 10 to 20% unmoderated).

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Benchmarking the User Experience (Sauro)
  • How to Conduct a Quantitative Usability Test (Sauro)
  • Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)

Also known as

  • Margin-of-error sample sizing
  • Sample size for comparison studies

Categories

  • Research Operations
  • Research Design & Rigor