Adjusted-Wald Confidence Interval for Completion Rates

Description

A confidence interval for a task success rate that stays accurate with the tiny samples typical of usability tests.

Completion rates from usability tests are binomial proportions measured on 5 to 20 people. The textbook Wald interval (p plus or minus z times the standard error) collapses at these sizes: it can run below 0% or above 100% and covers the true rate far less often than its nominal 95%. The adjusted-Wald interval fixes this by adding z²/2 successes and z² trials (about two successes and four attempts at 95%) before computing the interval. Sauro and Lewis showed it has the best average coverage of the common options for usability data, beating the exact Clopper-Pearson interval, which is too conservative. It turns "4 of 5 users succeeded" into an honest statement such as "between 36% and 98% of all users would likely succeed".

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Count successes x and attempts n for the task, using a pre-agreed success rule.
  2. Choose the confidence level (90% is common for formative work, 95% for summative claims) and look up z.
  3. Compute the adjusted proportion p_adj = (x + z²/2) / (n + z²).
  4. Compute the margin z × sqrt(p_adj(1 − p_adj) / (n + z²)) and form p_adj ± margin, truncating at 0 and 100%.
  5. Report the observed rate, the interval and n together, and plot intervals as error bars when comparing tasks.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • How to Conduct a Quantitative Usability Test (Sauro)
  • Benchmarking the User Experience (Sauro)

Also known as

  • Agresti-Coull interval
  • Adjusted Wald binomial interval

Categories

  • Analytics & Behaviour
  • Statistics & Modelling