Description
Programming a large language model, via its API, to apply a defined codebook or propose first-order codes for transcript segments, with structured outputs that analysts verify.
A qualitative coding agent wraps an LLM in a controlled procedure: a system prompt with the research question and codebook, segment-by-segment input with participant metadata, and outputs constrained to a JSON schema (code, rationale, verbatim evidence span, confidence). Run through the API rather than a chat window, it is repeatable, logged and scalable to hundreds of transcripts. It is most defensible for deductive coding against a well-defined codebook, and as a first-pass suggester in inductive work. Rigour comes from pinning the model version and temperature, requiring verbatim evidence for every code, measuring agreement with human coders (e.g. Cohen's kappa) on a held-out sample before scaling, logging all prompts and outputs, and keeping humans responsible for interpretation.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Active
- Goal
- Synthesise data
- Stage
- Discovery, Definition
- Data
- Qualitative
- Effort
- Weeks
Steps
- Write the research question and a codebook with definitions, inclusion and exclusion criteria and examples for each code.
- Segment transcripts (e.g. by speaker turn) and remove or anonymise personal data before any API call; check the data-processing terms.
- Define a JSON output schema requiring code IDs, a rationale and a verbatim quote span, and validate every response against it.
- Pin the model version and set temperature to 0 or low; log prompts, model ID and outputs per segment.
- Have two humans code a held-out sample of 10-20% of segments and compute agreement with the agent; proceed only if kappa is about 0.70 or higher for key codes.
- Run at scale, auto-check that quoted spans exist verbatim in the source, and route low-confidence or disputed segments to human review.
Sources
- Anthropic Claude API (claude-sonnet-4-20250514)
- structured outputs (JSON schema)
- asyncio
- Xiao, Z. et al. (2023) Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. IUI Companion
- Gao, J. et al. (2024) CollabCoder: A lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. CHI
- Saldaña, J. (2021) The Coding Manual for Qualitative Researchers, 4th ed.
- The Inquiry Series, C12 — AI Agents for Structured Qualitative Analysis at Scale
Also known as
- LLM-assisted qualitative coding
- AI coding assistant for transcripts
Categories
- Analytics & Behaviour
- Computational & Machine Learning