Building a qualitative coding agent (LLM API)

Description

Programming a large language model, via its API, to apply a defined codebook or propose first-order codes for transcript segments, with structured outputs that analysts verify.

A qualitative coding agent wraps an LLM in a controlled procedure: a system prompt with the research question and codebook, segment-by-segment input with participant metadata, and outputs constrained to a JSON schema (code, rationale, verbatim evidence span, confidence). Run through the API rather than a chat window, it is repeatable, logged and scalable to hundreds of transcripts. It is most defensible for deductive coding against a well-defined codebook, and as a first-pass suggester in inductive work. Rigour comes from pinning the model version and temperature, requiring verbatim evidence for every code, measuring agreement with human coders (e.g. Cohen's kappa) on a held-out sample before scaling, logging all prompts and outputs, and keeping humans responsible for interpretation.

Attributes

Loss type
Active
Effort
Weeks

Steps

  1. Write the research question and a codebook with definitions, inclusion and exclusion criteria and examples for each code.
  2. Segment transcripts (e.g. by speaker turn) and remove or anonymise personal data before any API call; check the data-processing terms.
  3. Define a JSON output schema requiring code IDs, a rationale and a verbatim quote span, and validate every response against it.
  4. Pin the model version and set temperature to 0 or low; log prompts, model ID and outputs per segment.
  5. Have two humans code a held-out sample of 10-20% of segments and compute agreement with the agent; proceed only if kappa is about 0.70 or higher for key codes.
  6. Run at scale, auto-check that quoted spans exist verbatim in the source, and route low-confidence or disputed segments to human review.

Sources

  • Anthropic Claude API (claude-sonnet-4-20250514)
  • structured outputs (JSON schema)
  • asyncio
  • Xiao, Z. et al. (2023) Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. IUI Companion
  • Gao, J. et al. (2024) CollabCoder: A lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. CHI
  • Saldaña, J. (2021) The Coding Manual for Qualitative Researchers, 4th ed.
  • The Inquiry Series, C12 — AI Agents for Structured Qualitative Analysis at Scale

Also known as

  • LLM-assisted qualitative coding
  • AI coding assistant for transcripts

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning