Description
Works out how many participants a benchmark or comparison needs to hit a target precision or detect a meaningful difference.
Summative studies measure rather than discover, so sample size follows from statistics, not the five-user rule. There are three situations. For a stand-alone benchmark, n depends on the margin of error you can accept around a completion rate or mean. For comparisons, n depends on the smallest difference worth detecting, the variability, the confidence level and the power, and within-subjects designs need far fewer people than between-subjects ones. For comparisons against an industry benchmark, n depends on how close to the benchmark you expect to be. Sauro and Lewis give iterative t-based formulas that work for small samples, and practical tables, such as roughly 93 people for a ±10 point margin on a completion rate.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Siloed
- Goal
- Support a decision
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Classify the study: stand-alone estimate, comparison (within or between), or test against a benchmark.
- Choose confidence level and, for comparisons, power (80% is typical).
- Get a variability estimate from past studies (SUS SD is about 17.7; for proportions use 0.5 if unknown).
- Apply the formula or table, iterating the t value until n stabilises.
- Inflate for expected drop-outs and data cleaning losses (often 10 to 20% unmoderated).
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Benchmarking the User Experience (Sauro)
- How to Conduct a Quantitative Usability Test (Sauro)
- Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)
Also known as
- Margin-of-error sample sizing
- Sample size for comparison studies
Categories
- Research Operations
- Research Design & Rigor