Description
Have colleagues or crowd workers tag thousands of short entries against a taxonomy, with a 'none of the above' option and double coding to measure agreement.
Large diary studies, feedback channels and open-ended surveys can produce thousands of short entries that no single researcher can code in time. This method distributes the coding. If a taxonomy exists (for a travel app, its features such as flight search or account settings), volunteers inside the organisation or paid crowd workers tag each entry with one label. If not, a provisional taxonomy is built by affinity diagramming a random subset, and taggers can choose 'none of the above' and propose a label, which reveals missing categories. Rigour comes from clear instructions with examples, assigning at least two taggers per item, computing inter-rater reliability such as Cohen's kappa, resolving disagreements and revising the taxonomy. It works only when entries stand alone and the domain is easy to grasp.
Attributes
- Discipline
- Qualitative Analysis & Synthesis
- Loss type
- Latent
- Stage
- Discovery, Post-launch
- Data
- Mixed
- Effort
- Days
Steps
- Define a taxonomy, or derive a provisional one by affinity diagramming a random subset of entries.
- Write tagging instructions with worked examples for each label.
- Distribute entries to internal volunteers or crowd workers after checking privacy and consent allow it.
- Offer a 'none of the above' option with a field for a suggested label.
- Assign at least two taggers to each entry and compute agreement, for example Cohen's kappa.
- Resolve disagreements and revise the taxonomy from suggested labels, then re-tag affected items.
Sources
- Understanding Your Users (Baxter et al., 2015)
- Computer Vision: A Modern Approach (Forsyth & Ponce, 2011)
Also known as
- Crowd coding
- Distributed tagging of diary entries
- Volunteer coding against a taxonomy
Categories
- User Research
- Qualitative Analysis & Synthesis