Cooperative Evaluation

Description

A relaxed think-aloud variant in which the participant acts as a collaborator who criticises the system, while evaluator and participant may question each other during tasks.

In standard think-aloud the participant is a subject and the evaluator stays silent. Cooperative evaluation reframes the participant as a co-evaluator of the system: they think aloud from the start and are invited to criticise. The evaluator may ask 'why' or 'what if' questions when behaviour is unclear, and the participant may ask the evaluator for clarification when stuck. Dix and colleagues list its advantages: the process is less constrained and easier for evaluators to learn, users are encouraged to criticise, and confusion is clarified the moment it occurs, which maximises the number of problems found. Its value rests on recording the session and analysing the resulting protocol, optionally with a post-task walkthrough. It was designed as a practical technique for designers without psychology training.

Attributes

Loss type
Renamed
Effort
Days

Steps

  1. Prepare realistic tasks and brief the participant that they are a collaborator evaluating the system, not the person being tested, and that criticism is wanted.
  2. Ask the participant to think aloud while doing each task.
  3. When behaviour is unclear, ask why or what-if questions rather than waiting silently.
  4. Answer the participant's requests for clarification when a problem arises, noting each point of confusion.
  5. Record audio, video and screen so the session forms a protocol.
  6. Analyse the protocol for problem areas and the user's explanations, adding a post-task walkthrough of the recording if needed.

Sources

  • Human-Computer Interaction (Dix et al., 2004)

Also known as

  • Co-operative evaluation
  • Monk cooperative evaluation

Categories

  • User Research
  • Usability & Evaluation