Description
Instead of timing set tasks, let domain experts explore real data freely and count, time and rate the insights they report.
Task-time studies fit visualisation tools poorly: analysis is long and high level, users view data from many angles, questions emerge during exploration, and important discoveries are rare. Insight-based evaluation measures what visualisation is for. Domain experts receive realistic data from their own field and a tool, list their initial questions, then explore freely while thinking aloud for a fixed period. Every reported insight is recorded and timestamped, then coded for category, correctness, depth (breadth versus deep or hypothesis-generating) and domain value, ideally by independent experts. Tools or versions are compared on number of insights, their value and time to first insight. Shneiderman and colleagues present it as a first step beyond controlled tasks, complementing long-term case studies of real analysts.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Siloed
- Stage
- Evaluation
- Data
- Mixed
- Effort
- Weeks
Steps
- Recruit domain experts and prepare realistic data from their field.
- Ask each participant to list initial questions of interest.
- Let them explore freely with the tool while thinking aloud for a fixed period.
- Record each insight with a timestamp.
- Code insights for correctness, depth, category and domain value, ideally rated by independent experts.
- Compare tools or versions on insight count, value and time to first insight.
Sources
- Designing the User Interface (Shneiderman et al., 2018)
Also known as
- Insight-based evaluation of visualisations
- Insight-based study
- Insight counting
- Open-ended visualisation evaluation
Categories
- User Research
- Usability & Evaluation