Problem Discovery Sample Size Model (1 − (1 − p)^n)

Description

Plans how many users a formative test needs to find problems of a given frequency, and checks how many remain unseen.

The famous "five users find 85% of problems" claim is a special case of the cumulative binomial model: the chance of seeing a problem at least once in n users is 1 − (1 − p)^n, where p is how often that problem affects users. Five users find 85% of problems only if the average p is about 0.31. Lewis showed that p estimated from a small test is inflated and corrected it by averaging a Good-Turing adjustment with a normalisation step. Used properly, the model turns sample size into a deliberate choice: "we want a 90% chance of seeing any problem that affects at least 15% of users", which needs 15 people. It also estimates how many problems the study probably missed.

Attributes

Loss type
Renamed
Stage
Evaluation
Data
Mixed
Effort
Hours

Steps

  1. Decide the smallest problem frequency p you care about and the discovery probability you want (for example p = 0.15 at 90%).
  2. Solve n = ln(1 − goal) / ln(1 − p) and round up.
  3. Run the test and record a participant-by-problem 0/1 matrix.
  4. Estimate p from the matrix and apply Lewis's adjustment (average of Good-Turing and normalisation) to deflate it.
  5. Compute the expected proportion of problems found and the likely number still undiscovered; decide whether to run more sessions.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • How to Conduct a Quantitative Usability Test (Sauro)

Also known as

  • Five-user rule
  • Cumulative binomial problem discovery
  • Good-Turing adjusted discovery

Categories

  • User Research
  • Usability & Evaluation