Description
After each task, rate confidence of success; failures given the top confidence rating count as disasters, reported per task as the rate of users who failed without knowing.
A post-task item asks participants how confident they are that they completed the task successfully, usually on a seven-point scale (not to be confused with a statistical confidence interval). Confidence alone is a task metric that can be compared across tasks and versions, and it can feed a summarised usability metric. Crossed with binary completion it gives the disaster rate: the share of participants who failed but chose the top confidence point. Sauro treats this as worse than ordinary failure, because people leave believing an order went through or carrying wrong information, so the problem never reaches support and is invisible in satisfaction data. The rigour depends on precise, predefined success criteria and on a consistent scale, since a five-point scale makes the top category looser.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Latent
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Define success criteria for every task before testing so each attempt can be coded as success or failure without debate.
- After each task ask a seven-point confidence item (five, nine or eleven points are possible but change the meaning of the top point) about having completed the task successfully.
- Report mean confidence per task with confidence intervals, alongside completion rate.
- Count failed attempts that received the top scale point and divide by the number of participants attempting the task to get the disaster rate.
- Flag tasks with high disaster rates and review the recordings or open answers to find what misled participants.
- Track confidence and disaster rate across benchmark rounds and competitor products.
Sources
- Benchmarking the User Experience (Sauro, 2018)
Also known as
- Disaster rate
- Confident failure rate
- Post-task confidence rating
- Disasters
Categories
- User Research
- Usability & Evaluation