Description
Catalogue every usability issue from test data, merge, count and prioritise them, and report metrics such as unique issues by severity, issues per participant and percentage affected.
Turning raw qualitative test notes into a prioritised issue catalogue is a disciplined sequence: get to know the data, list every issue per transcript without categorising or counting, group similar issues, return to recordings to separate subtly different problems, count instances, and weigh context, because a rare issue may still be critical, especially if the moderator had to assist. The catalogue records issue number, priority, description, recommendation and cost of fixing; a journey-board variant lays issues under each step of the journey. Albert and Tullis add metrics: unique issues per iteration (24, 12, then 4 across three rounds), split by severity; mean issues per participant with confidence interval; the percentage of participants hitting each issue; issues by three to eight categories and by task; and the percentage meeting any high-severity issue. Counts depend on evaluators and tasks, so severity schemes must stay consistent across studies.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Active
- Stage
- Evaluation
- Data
- Mixed
- Effort
- Days
Steps
- Review data after each session, then list every issue or need per transcript without categorising or counting.
- Group similar issues and return to recordings to decide whether differently described problems are the same; merge duplicates across participants and studies.
- Log each unique issue with participants, task, category (three to eight categories) and severity using one consistent scheme.
- Count instances per issue and weigh context: severity, whether help was needed, and whether a rare issue would affect others.
- Prioritise in a catalogue table with issue number, priority, description, recommendation and cost of fixing, or on a journey board with one column per step.
- Report unique issues per iteration by severity, issues per participant (mean and confidence interval), percentage of participants affected per issue and percentage meeting any high-severity issue; with small samples report frequencies rather than percentages.
Sources
- Measuring the User Experience (Albert & Tullis, 2022)
- User Research (Marsh, 2022)
- User Research: A Practical Guide to Designing Better Products and Services (Marsh, 2018)
- Customer Analytics for Dummies (Sauro, 2015)
Also known as
- Issues-based metrics
- Usability issue metrics
- Issue frequency and severity analysis
- Usability issue cataloguing
- Issue catalogue
- Journey board analysis
Categories
- User Research
- Usability & Evaluation