Monte Carlo Evaluator-Aggregation Curve

Description

Estimate how many evaluators you need by resampling random groups from an evaluator-by-problem matrix and plotting the share of known problems each group size finds.

Nielsen and Molich collected independent heuristic-evaluation reports from 34 to 77 evaluators on four interfaces, scored them against a master list of known problems and recorded a binary evaluator-by-problem matrix. For every group size they drew between five and nine thousand random sets of evaluators, took the union of their problems and averaged the proportion found. The four curves had a common shape: a steep rise to about five evaluators, flattening to ten. Random rather than best evaluators were used because organisations must use available staff. Sorting the matrix by evaluator success and problem ease shows that detection is not cumulative, which is why aggregation pays. Later critiques note the curve is relative to known problems only, assumes perfect merging and consensus on false positives, and depends heavily on evaluator expertise.

Attributes

Loss type
Latent
Stage
Evaluation
Effort
Days

Steps

  1. Have a pool of evaluators (ideally 20 or more) independently evaluate the same interface.
  2. Build a master list of known problems and revise it after a first pass, since evaluators find problems the researchers missed.
  3. Score each report against the list and record a binary evaluator-by-problem matrix; plot it sorted by evaluator success and problem ease.
  4. For each group size from 1 to the pool size, draw thousands of random groups (Nielsen and Molich used 5,000 to 9,000) with replacement.
  5. Take the union of each group's problems and compute the proportion of known problems found.
  6. Average across draws, plot mean proportion against group size and tabulate selected sizes.
  7. Choose the evaluator count where marginal gain flattens (often 3 to 5) and report that the curve is relative to known problems only.

Sources

  • Heuristic Evaluation of User Interfaces: Exploration and Evaluation (Ó Broin, 2011)
  • Heuristic Evaluation of User Interfaces (Nielsen & Molich, 1990)

Also known as

  • Monte Carlo aggregation of evaluator detections
  • Hypothetical aggregation curve
  • Random aggregate resampling of evaluator-by-problem matrix
  • Empirical problem-discovery curve

Categories

  • User Research
  • Usability & Evaluation