Designer versus Observed Task Model Comparison

Description

Compare the designer's expected task model with task models inferred from logged field use, annotate errors and unforeseen tasks, and classify deviations to drive redesign.

Exploratory software such as open learning environments supports many strategies, so a single expected task flow is a poor yardstick. This method writes down the designer's task model of expected strategies, then reconstructs each real user's task model from keystroke-level logs and video gathered in the field. Observed models are annotated with plan operators and markers for errors, repetitions and tasks the designer never foresaw, and every deviation is classified into one of five categories of unexpected behaviour. Comparing designer and observed models shows where the design's assumptions break, which unforeseen strategies deserve support and which errors are systematic. A prior heuristic evaluation focuses the field study, and the cycle repeats in a spiral of redesign. Its rigour comes from making both models explicit in the same notation, so mismatches are traceable rather than impressionistic.

Attributes

Loss type
Latent
Data
Mixed
Effort
Weeks

Steps

  1. Write the designer's task model of expected strategies, for example as a hierarchical task model with plan operators.
  2. Run a heuristic evaluation to identify likely problem areas and focus the field study.
  3. Log keystroke-level interaction and record video of users in their real setting.
  4. Infer each user's actual task model from the logs and video, annotating errors, repetitions and unforeseen tasks.
  5. Classify each deviation from the designer's model into the five categories of unexpected behaviour.
  6. Compare designer and observed models, revise the design and repeat the cycle.

Sources

  • Handbook of Cognitive Task Design (Hollnagel, 2003)

Also known as

  • Designer versus observed user task model in field evaluation
  • Task model deviation analysis

Categories

  • User Research
  • Usability & Evaluation