Description
Scan session data for eight observable signs that a usability problem occurred, from negative emotion and surprise to unnoticed omissions, overload and trust failures.
Defining a usability problem as 'a needed interface change' is too vague to apply consistently, which contributes to the evaluator effect. Hertzum lists eight indicators to look for in session data: negative emotion such as frustration or uncertainty, verbal or bodily; surprise; need for assistance; trying several actions and backtracking; unnoticed omission of information or steps, visible only by comparison with the correct solution; mental overload, often signalled by fewer verbalisations; violation of standards or conventions; and failure to instil trust and credibility. No universal threshold exists: smaller difficulties count on central features. Some teams require at least two users before recording a problem; with small samples, single-user problems should count. Cultural differences matter, since indirectness can hide dissatisfaction, so culturally matched evaluators help.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Vestigial
- Stage
- Evaluation
- Data
- Qualitative
- Effort
- Days
Steps
- Agree the eight indicators and a solution key per task before analysis.
- Analyse each session individually, marking episodes that match any indicator, using behaviour as well as speech.
- Compare task outcomes with the solution key to catch unnoticed omissions.
- Judge whether each difficulty is significant given the feature's centrality.
- Set a single-user rule appropriate to sample size and apply it consistently.
- Account for cultural expression, ideally with culturally matched evaluators, then merge episodes across sessions into a candidate problem list.
Sources
- Usability Testing: A Practitioner's Guide (Hertzum, 2020)
Also known as
- Problem indicators
- Breakdown indicators in usability tests
Categories
- User Research
- Usability & Evaluation