Description
Applying a human-built codebook to large transcript corpora with an LLM, validated against human coding with kappa thresholds and human review of low-confidence output.
Large language models can apply qualitative codes at a scale no team can match, but they also produce plausible yet wrong labels. This method keeps theory and interpretation human. Researchers first build a codebook with definitions, inclusion and exclusion rules and examples, then human-code 20 to 25 percent of the corpus as a test set. The model is prompted to return structured output (code, supporting quote, confidence, rationale) for each segment, and human-AI agreement is computed; deployment proceeds only above a threshold such as kappa 0.70, with low agreement treated as evidence of codebook ambiguity. The model codes the rest, with periodic human spot checks and review of low-confidence segments. Humans produce the themes and conclusions.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Active
- Stage
- Discovery, Definition
- Data
- Qualitative
- Effort
- Days
Steps
- Develop the coding manual first: Code definitions, inclusion/exclusion criteria, and worked examples — exactly as for human coding.
- Structured JSON output schema: {'{'}code_name, supporting_quote, confidence_score, rationale{'}'}. Enables systematic comparison with human codes.
- Human baseline coding first: Human coders code 20–25% of the corpus. This becomes the IRR test set.
- Calculate IRR human vs. AI: κ > 0.70 before deploying the AI on the remaining corpus. If κ < 0.70, revise coding instructions — the failure reveals ambiguity in your scheme.
- AI codes remaining corpus with human spot-checking every 10th transcript. Flag segments with confidence < 0.7 for human review.
- Human interpretation: The AI produces labelled segments. Humans produce themes, propositions, and the final analysis.
Sources
- AI for Humans (Heaton)
- Anthropic Claude API (claude-sonnet-4-20250514)
- Sage Handbook of Qualitative Data Analysis (Flick)
- Content Analysis: An Introduction to Its Methodology (Krippendorff)
Also known as
- LLM-assisted coding
- AI coding agent
- Human-defined codebook applied by LLM
Categories
- Analytics & Behaviour
- Computational & Machine Learning