Comparative Usability Evaluation (CUE) Studies

Description

Have independent professional teams evaluate the same product or session videos and report issues, then measure how much their findings overlap.

CUE studies test the reproducibility of usability evaluation itself. Several teams independently evaluate one product with their usual methods, or, as in CUE-9, all watch the same session recordings, and report issues in a common format. Reports are merged into a master list of unique issues and the overlap is computed. Results have been sobering: in CUE-1 only 1 of 141 issues was found by all four teams; in CUE-2, 75% of issues were found by only one of nine teams; in CUE-9, 35 professionals watching the same five 30-minute videos reported 223 unique issues, only four reported by at least 75% of them, with 40% reported by one evaluator alone. Hertzum, Molich and Jacobsen traced this evaluator effect to human judgement in detecting, rating and reporting issues. The same design can audit an organisation's own evaluators.

Attributes

Loss type
Siloed
Stage
Evaluation
Data
Mixed
Effort
Weeks

Steps

  1. Recruit several independent evaluators or teams.
  2. Give all the same product and scenario, or the same session recordings to remove differences in test sessions.
  3. Have each report issues independently in a common format with severity ratings.
  4. Merge reports into a master list of unique issues, matching duplicates with explicit rules.
  5. Compute how many evaluators reported each issue and the share found by only one.
  6. Analyse reasons for disagreement and set policies, such as using multiple evaluators for critical studies.

Sources

  • Measuring the User Experience (Albert & Tullis, 2022)
  • Improving a Human-Computer Dialogue (Molich & Nielsen, 1990)

Also known as

  • Comparative Usability Evaluation (CUE) multi-team studies
  • CUE studies
  • Evaluator-effect studies
  • Molich CUE

Categories

  • Research Operations
  • Research Design & Rigor