AI-Assisted Qualitative Coding

Description

Applying a human-built codebook to large transcript corpora with an LLM, validated against human coding with kappa thresholds and human review of low-confidence output.

Large language models can apply qualitative codes at a scale no team can match, but they also produce plausible yet wrong labels. This method keeps theory and interpretation human. Researchers first build a codebook with definitions, inclusion and exclusion rules and examples, then human-code 20 to 25 percent of the corpus as a test set. The model is prompted to return structured output (code, supporting quote, confidence, rationale) for each segment, and human-AI agreement is computed; deployment proceeds only above a threshold such as kappa 0.70, with low agreement treated as evidence of codebook ambiguity. The model codes the rest, with periodic human spot checks and review of low-confidence segments. Humans produce the themes and conclusions.

Attributes

Loss type
Active
Effort
Days

Steps

  1. Develop the coding manual first: Code definitions, inclusion/exclusion criteria, and worked examples — exactly as for human coding.
  2. Structured JSON output schema: {'{'}code_name, supporting_quote, confidence_score, rationale{'}'}. Enables systematic comparison with human codes.
  3. Human baseline coding first: Human coders code 20–25% of the corpus. This becomes the IRR test set.
  4. Calculate IRR human vs. AI: κ > 0.70 before deploying the AI on the remaining corpus. If κ < 0.70, revise coding instructions — the failure reveals ambiguity in your scheme.
  5. AI codes remaining corpus with human spot-checking every 10th transcript. Flag segments with confidence < 0.7 for human review.
  6. Human interpretation: The AI produces labelled segments. Humans produce themes, propositions, and the final analysis.

Sources

  • AI for Humans (Heaton)
  • Anthropic Claude API (claude-sonnet-4-20250514)
  • Sage Handbook of Qualitative Data Analysis (Flick)
  • Content Analysis: An Introduction to Its Methodology (Krippendorff)

Also known as

  • LLM-assisted coding
  • AI coding agent
  • Human-defined codebook applied by LLM

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning