Insight-Based Evaluation

Description

Instead of timing set tasks, let domain experts explore real data freely and count, time and rate the insights they report.

Task-time studies fit visualisation tools poorly: analysis is long and high level, users view data from many angles, questions emerge during exploration, and important discoveries are rare. Insight-based evaluation measures what visualisation is for. Domain experts receive realistic data from their own field and a tool, list their initial questions, then explore freely while thinking aloud for a fixed period. Every reported insight is recorded and timestamped, then coded for category, correctness, depth (breadth versus deep or hypothesis-generating) and domain value, ideally by independent experts. Tools or versions are compared on number of insights, their value and time to first insight. Shneiderman and colleagues present it as a first step beyond controlled tasks, complementing long-term case studies of real analysts.

Attributes

Loss type
Siloed
Stage
Evaluation
Data
Mixed
Effort
Weeks

Steps

  1. Recruit domain experts and prepare realistic data from their field.
  2. Ask each participant to list initial questions of interest.
  3. Let them explore freely with the tool while thinking aloud for a fixed period.
  4. Record each insight with a timestamp.
  5. Code insights for correctness, depth, category and domain value, ideally rated by independent experts.
  6. Compare tools or versions on insight count, value and time to first insight.

Sources

  • Designing the User Interface (Shneiderman et al., 2018)

Also known as

  • Insight-based evaluation of visualisations
  • Insight-based study
  • Insight counting
  • Open-ended visualisation evaluation

Categories

  • User Research
  • Usability & Evaluation