Description
Plans how many users a formative test needs to find problems of a given frequency, and checks how many remain unseen.
The famous "five users find 85% of problems" claim is a special case of the cumulative binomial model: the chance of seeing a problem at least once in n users is 1 − (1 − p)^n, where p is how often that problem affects users. Five users find 85% of problems only if the average p is about 0.31. Lewis showed that p estimated from a small test is inflated and corrected it by averaging a Good-Turing adjustment with a normalisation step. Used properly, the model turns sample size into a deliberate choice: "we want a 90% chance of seeing any problem that affects at least 15% of users", which needs 15 people. It also estimates how many problems the study probably missed.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Renamed
- Stage
- Evaluation
- Data
- Mixed
- Effort
- Hours
Steps
- Decide the smallest problem frequency p you care about and the discovery probability you want (for example p = 0.15 at 90%).
- Solve n = ln(1 − goal) / ln(1 − p) and round up.
- Run the test and record a participant-by-problem 0/1 matrix.
- Estimate p from the matrix and apply Lewis's adjustment (average of Good-Turing and normalisation) to deflate it.
- Compute the expected proportion of problems found and the likely number still undiscovered; decide whether to run more sessions.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- How to Conduct a Quantitative Usability Test (Sauro)
Also known as
- Five-user rule
- Cumulative binomial problem discovery
- Good-Turing adjusted discovery
Categories
- User Research
- Usability & Evaluation