Heuristic Evaluation with Inter-Evaluator Reliability

Description

Heuristic evaluation as originally specified: several independent evaluators, aggregation of all unique problems, and measured overlap between evaluators to judge coverage and reliability.

Heuristic evaluation has evaluators inspect an interface against a set of usability heuristics. Nielsen and Molich found that single evaluators detect only about 20-50% of problems, so the method's validity depends on several evaluators (typically three to five) working independently and on aggregating every unique problem found by anyone, not averaging or voting. Evaluator independence must be preserved until individual reports are complete. Reliability is assessed by computing the overlap between evaluators (for example, any-two agreement) and how many problems were found by only one person, which indicates how far from saturation the evaluation is. Severity is rated afterwards, ideally independently. In current practice heuristic evaluation is often a single designer's review, which keeps the name but loses the statistical basis of the method.

Attributes

Loss type
Vestigial
Stage
Evaluation
Effort
Days

Steps

  1. 3–5 evaluators: each independently evaluates the interface against all 10 heuristics.
  2. No discussion during individual phases.
  3. After all have finished: aggregate lists.
  4. Count unique problems.
  5. Measure evaluator reliability: Fleiss' κ on binary problem detection (did evaluator i find problem j?).
  6. Low κ (< 0.40) means evaluators are finding different problems — discuss and align on heuristic interpretation before proceeding.

Sources

  • Usability Engineering — Jakob Nielsen
  • Hertzum & Jacobsen, The evaluator effect (International Journal of Human-Computer Interaction, 2001)

Also known as

  • Multi-evaluator heuristic evaluation
  • Nielsen-Molich heuristic evaluation

Categories

  • User Research
  • Usability & Evaluation