Task Success Coding and Error-Opportunity Rates

Description

Rules for scoring each task attempt as a success or failure and counting errors against the number of chances to make them.

Completion rates and error counts are only comparable when they are coded consistently. Sauro's guidance defines success in advance (a correct end state, ideally checkable by URL or answer), treats assisted completions as failures, and codes partial success only with explicit criteria. For errors, raw counts are misleading because a twelve-field form offers more chances to slip than a two-click task. Dividing errors by error opportunities, the places a user could reasonably go wrong, gives an error rate that can be compared across tasks and products, and the proportion of error-free attempts feeds into SUM. Published data put the average completion rate around 78% and suggest about 0.7 errors per task is typical.

Attributes

Loss type
Vestigial
Stage
Evaluation
Effort
Hours

Steps

  1. Write a success criterion for each task before testing, including what counts as assistance.
  2. Code each attempt as 1 (success) or 0 (failure), never as a percentage of steps.
  3. Define error opportunities by walking the ideal path and listing each point where a slip or mistake is possible.
  4. Log each error with its type and step during sessions or from recordings.
  5. Compute errors per task, error rate per opportunity and the proportion of error-free attempts.

Sources

  • How to Conduct a Quantitative Usability Test (Sauro)
  • Benchmarking the User Experience (Sauro)
  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • Task completion coding
  • Error rate per opportunity

Categories

  • User Research
  • Usability & Evaluation