Usability Problem Identification Criteria

Description

Scan session data for eight observable signs that a usability problem occurred, from negative emotion and surprise to unnoticed omissions, overload and trust failures.

Defining a usability problem as 'a needed interface change' is too vague to apply consistently, which contributes to the evaluator effect. Hertzum lists eight indicators to look for in session data: negative emotion such as frustration or uncertainty, verbal or bodily; surprise; need for assistance; trying several actions and backtracking; unnoticed omission of information or steps, visible only by comparison with the correct solution; mental overload, often signalled by fewer verbalisations; violation of standards or conventions; and failure to instil trust and credibility. No universal threshold exists: smaller difficulties count on central features. Some teams require at least two users before recording a problem; with small samples, single-user problems should count. Cultural differences matter, since indirectness can hide dissatisfaction, so culturally matched evaluators help.

Attributes

Loss type
Vestigial
Stage
Evaluation
Effort
Days

Steps

  1. Agree the eight indicators and a solution key per task before analysis.
  2. Analyse each session individually, marking episodes that match any indicator, using behaviour as well as speech.
  3. Compare task outcomes with the solution key to catch unnoticed omissions.
  4. Judge whether each difficulty is significant given the feature's centrality.
  5. Set a single-user rule appropriate to sample size and apply it consistently.
  6. Account for cultural expression, ideally with culturally matched evaluators, then merge episodes across sessions into a candidate problem list.

Sources

  • Usability Testing: A Practitioner's Guide (Hertzum, 2020)

Also known as

  • Problem indicators
  • Breakdown indicators in usability tests

Categories

  • User Research
  • Usability & Evaluation