Crowdsourced Tagging of Qualitative Data

Description

Have colleagues or crowd workers tag thousands of short entries against a taxonomy, with a 'none of the above' option and double coding to measure agreement.

Large diary studies, feedback channels and open-ended surveys can produce thousands of short entries that no single researcher can code in time. This method distributes the coding. If a taxonomy exists (for a travel app, its features such as flight search or account settings), volunteers inside the organisation or paid crowd workers tag each entry with one label. If not, a provisional taxonomy is built by affinity diagramming a random subset, and taggers can choose 'none of the above' and propose a label, which reveals missing categories. Rigour comes from clear instructions with examples, assigning at least two taggers per item, computing inter-rater reliability such as Cohen's kappa, resolving disagreements and revising the taxonomy. It works only when entries stand alone and the domain is easy to grasp.

Attributes

Loss type
Latent
Data
Mixed
Effort
Days

Steps

  1. Define a taxonomy, or derive a provisional one by affinity diagramming a random subset of entries.
  2. Write tagging instructions with worked examples for each label.
  3. Distribute entries to internal volunteers or crowd workers after checking privacy and consent allow it.
  4. Offer a 'none of the above' option with a field for a suggested label.
  5. Assign at least two taggers to each entry and compute agreement, for example Cohen's kappa.
  6. Resolve disagreements and revise the taxonomy from suggested labels, then re-tag affected items.

Sources

  • Understanding Your Users (Baxter et al., 2015)
  • Computer Vision: A Modern Approach (Forsyth & Ponce, 2011)

Also known as

  • Crowd coding
  • Distributed tagging of diary entries
  • Volunteer coding against a taxonomy

Categories

  • User Research
  • Qualitative Analysis & Synthesis