Description
A confidence interval for a task success rate that stays accurate with the tiny samples typical of usability tests.
Completion rates from usability tests are binomial proportions measured on 5 to 20 people. The textbook Wald interval (p plus or minus z times the standard error) collapses at these sizes: it can run below 0% or above 100% and covers the true rate far less often than its nominal 95%. The adjusted-Wald interval fixes this by adding z²/2 successes and z² trials (about two successes and four attempts at 95%) before computing the interval. Sauro and Lewis showed it has the best average coverage of the common options for usability data, beating the exact Clopper-Pearson interval, which is too conservative. It turns "4 of 5 users succeeded" into an honest statement such as "between 36% and 98% of all users would likely succeed".
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Count successes x and attempts n for the task, using a pre-agreed success rule.
- Choose the confidence level (90% is common for formative work, 95% for summative claims) and look up z.
- Compute the adjusted proportion p_adj = (x + z²/2) / (n + z²).
- Compute the margin z × sqrt(p_adj(1 − p_adj) / (n + z²)) and form p_adj ± margin, truncating at 0 and 100%.
- Report the observed rate, the interval and n together, and plot intervals as error bars when comparing tasks.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- How to Conduct a Quantitative Usability Test (Sauro)
- Benchmarking the User Experience (Sauro)
Also known as
- Agresti-Coull interval
- Adjusted Wald binomial interval
Categories
- Analytics & Behaviour
- Statistics & Modelling