Single Usability Metric (SUM)

Description

Combines completion, time, errors and satisfaction for a task into one standardised usability score.

Stakeholders want one number; usability has several. SUM standardises each task metric against a specification limit and averages them. Completion is already a proportion. Task time is converted to the probability of finishing under a target time (using a z-score of log times). Errors become the proportion of error opportunities avoided. Post-task satisfaction is converted to the probability of exceeding a target rating. The result is a percentage between 0 and 100 per task, with a confidence interval, that can be rolled up across tasks. Principal components work by Sauro and Kindlund showed the four metrics share enough variance to justify combining them, while still being worth inspecting separately.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Collect completion, time, errors (with error opportunities) and a post-task rating for each task.
  2. Set specification limits: a target time (for example from expert times or goals) and a target satisfaction score.
  3. Convert each metric to a percentage: success rate; proportion under target time; proportion of opportunities error-free; proportion above satisfaction target.
  4. Average the percentages per task, compute an interval, and average across tasks for a product-level SUM.
  5. Report SUM with the component metrics so a low score can be diagnosed.

Sources

  • How to Conduct a Quantitative Usability Test (Sauro)
  • Benchmarking the User Experience (Sauro)
  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • SUM score

Categories

  • User Research
  • Usability & Evaluation