Description
Researchers code a sample of sessions, tickets or transcripts against an explicit codebook to create reliable labels for training and validating classifiers.
Classifiers inherit the quality of their labels. This method treats labelling as qualitative research: build a codebook with operational definitions and examples of each state (e.g. frustrated, confused, exploring), train coders, double-code a substantial sample, measure inter-rater reliability and resolve disagreements before labels are used. Labelling from session replays or think-aloud data captures meaning that event logs miss, and the disagreements themselves often reveal ambiguous states worth studying. Rigour comes from a stratified sample covering edge cases, blind double coding, reliability thresholds, documented adjudication and auditing label drift over time.
Attributes
- Discipline
- Qualitative Analysis & Synthesis
- Loss type
- Latent
- Stage
- Definition, Post-launch
- Data
- Qualitative
- Effort
- Weeks
Steps
- Draft a codebook with definitions, inclusion and exclusion rules, and anchor examples for each label.
- Draw a stratified sample of sessions, oversampling suspected rare states, typically 200-500 for initial modelling.
- Train 2-3 coders on 20-30 practice sessions and refine the codebook.
- Double-code at least 20% of the sample blind and compute Cohen's kappa or Krippendorff's alpha, targeting 0.70 or above (0.80 for high-stakes uses).
- Adjudicate disagreements, update the codebook, and record label confidence for ambiguous cases.
- Re-audit a fresh sample periodically to detect drift in user behaviour or coder interpretation.
Sources
- Pattern Classification (Richard Duda, Peter Hart & David Stork)
- Practical Statistics for Data Scientists (Peter Bruce, Andrew Bruce & Peter Gedeck)
- Methods Atlas (Vivek Gopinath)
- Krippendorff, Content Analysis: An Introduction to Its Methodology
- Pattern Classification (Duda)
- Practical Statistics for Data Scientists
- Methods Atlas (Gopinath)
- The Inquiry Series, Ch 54 — Linear Discriminant Analysis & Behavioral Classification
Also known as
- Qualitative annotation for ML
- Expert labelling of sessions
- Gold-standard coding
Categories
- User Research
- Qualitative Analysis & Synthesis