LLMs as qualitative analysis tools and their limits

Description

Uses large language models to assist coding, summarising and theme development in qualitative data, with explicit validation against human coding and safeguards against fabrication.

LLMs can apply codebooks to thousands of excerpts, propose candidate themes, summarise transcripts and retrieve quotes, making scale analysis possible. Their limits are equally important: hallucinated or misattributed quotes, flattening of minority and contradictory views, sensitivity to prompt wording, bias toward plausible-sounding generic themes, and opaque reasoning. Rigorous use treats the LLM as a coder whose reliability must be measured against humans, keeps humans responsible for interpretation, verifies every quote against the source, logs prompts and model versions, and handles data under privacy and consent constraints. Deductive coding with a clear codebook is the most reliable use; inductive theme generation needs the most scrutiny.

Attributes

Loss type
Active
Effort
Days

Steps

  1. Confirm consent and data governance permit processing transcripts with the chosen model, and redact personal data.
  2. Develop a codebook through human analysis of a subset of the data first.
  3. Prompt the LLM to apply the codebook with justifications and verbatim excerpts, fixing model version and temperature (e.g. 0).
  4. Compare LLM codes with human codes on at least 100-200 excerpts and compute Cohen's kappa; require 0.7 or above per code before relying on it.
  5. Verify every quoted excerpt programmatically against source text to catch fabrication.
  6. Use LLM-proposed themes as prompts for human interpretation, explicitly searching for disconfirming and minority cases.
  7. Document prompts, versions, agreement statistics and human decisions in the analysis audit trail.

Sources

  • A Practical Guide to Sentiment Analysis (Erik Cambria et al.)
  • Artificial Intelligence for Humans (Jeff Heaton)
  • Pattern Recognition and Machine Learning (Christopher Bishop)
  • Qualitative Data Analysis with LLMs: guidance from emerging literature (e.g. Xiao et al., 2023, Supporting qualitative analysis with large language models)
  • Practical Guide to Sentiment Analysis
  • AI for Humans (Heaton)
  • Pattern Recognition (Bishop)
  • The Inquiry Series, Ch 55 — Natural Language at Scale — Computational Qualitative Analysis

Also known as

  • LLM-assisted qualitative coding
  • AI-assisted thematic analysis

Categories

  • User Research
  • Qualitative Analysis & Synthesis