Non-Criterion Task Flagging and Source-of-Error Analysis

Description

Flag test tasks that fall below a preset success criterion, then attribute every observed error to the product component or components responsible, with primary and secondary causes.

Rubin and Chisnell describe this as the move from task orientation to product orientation in analysing a usability test. Each task has a success criterion and time benchmark set in advance: about 70 percent of participants for a typical early assessment test, rising towards 95 percent across iterations. Tasks below criterion are flagged. Errors are defined beforehand as divergences from expected behaviour, and then, for every error by every participant, the analyst identifies the responsible component or combination of components, separating primary from secondary culprits. Notes, logs, recordings, other observers and participant backgrounds are used for hard cases, ideally in a workshop with observers. Fixes are deliberately withheld until sources are known. The rigour lies in exhaustive, per-error attribution rather than a list of memorable problems, which explains why tasks fail and where differences between compared versions come from.

Attributes

Loss type
Vestigial
Stage
Evaluation
Data
Mixed
Effort
Days

Steps

  1. Before testing, set a success criterion and a time benchmark for each task (about 70 percent completion for an early assessment test, higher for later iterations).
  2. Define what counts as an error, using pilot sessions if needed; for summative tests fix the definitions before the study.
  3. After summarising performance, flag every task whose completion falls below its criterion.
  4. For each error by each participant on flagged tasks, identify the responsible component or combination, noting primary and secondary culprits.
  5. Resolve hard cases by reviewing recordings, logs, observer notes and the participant's background.
  6. Run the attribution as a workshop with observers where possible, and do not propose fixes until sources are identified.
  7. Report a three-column table of task, source of error and recommendation, and feed it into criticality ranking.

Sources

  • Handbook of Usability Testing (Rubin & Chisnell, 2008)

Also known as

  • Source of error analysis
  • Task-to-component error attribution
  • 70 percent success criterion

Categories

  • User Research
  • Usability & Evaluation