LLM Coding Replicability Test

Description

Code the same responses with an LLM in repeated independent runs under controlled temperature and context settings, and score how similar the code sets are, to choose replicable settings.

An LLM coding pipeline is a research instrument and should show reliability before its output is trusted. DeSutter operationalises reliability as running the same coding twice and comparing. In his study, 453 open-ended survey responses were coded twice per condition in a design crossing temperature (0 versus 1.0) with four context conditions: no context, a rolling list of codes applied so far, a fixed code list from a sample, and themes derived from the whole corpus first. Run-to-run similarity per response was scored with several metrics: translation measures (ROUGE-N, BLEU), set overlap (Jaccard, Dice-Sorensen) and semantic similarity (embedding cosine, Word Mover's Distance). Low temperature was more consistent, and the rolling codebook, a strategy that helps human analysts, was least consistent. Consistency is not accuracy, so agreement with human codes must be checked afterwards.

Attributes

Loss type
Latent
Stage
Evaluation
Data
Mixed
Effort
Days

Steps

  1. Assemble responses in a table and write a prompt template: instructions, code context, the response, output format and examples.
  2. Choose design factors that may affect variability, such as temperature and how much codebook context the model sees.
  3. For each condition run the full coding pipeline at least twice independently, logging model version and settings.
  4. For each response, compare the code sets from the runs with several metrics: ROUGE-N or BLEU, Jaccard or Dice-Sorensen, and embedding similarity.
  5. Check that metrics correlate, then compare consistency across conditions (with unequal variances, a Kruskal-Wallis test with pairwise comparisons).
  6. Adopt the most replicable settings, then check agreement with human codes on a sample before using the output.

Sources

  • The Stochastic Parrot's Song (DeSutter, 2024)

Also known as

  • Run-to-run consistency testing of LLM coding
  • LLM coding reliability experiment
  • Did you run it twice? test
  • LLM coding replicability test (run-to-run consistency of LLM qualitative coding)

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning