Appropriateness assessment for AI-assisted coding

Description

A structured judgement of whether AI coding fits a given study, based on epistemology, data sensitivity, task type and validation capacity, before any automation is used.

Not all qualitative work suits AI assistance. Deductive coding against a clear codebook, large corpora, and descriptive content analysis are better fits; reflexive thematic analysis, interpretative phenomenological analysis and work where researcher positionality is central are poor fits, because coding there is the researcher's own interpretive engagement. Data sensitivity, consent terms, participant vulnerability and language also matter, as does whether the team can validate outputs. A structured assessment makes the decision explicit and defensible, rather than defaulting to automation because tools are available. Rigour comes from documenting criteria, the decision and its rationale, and revisiting it if validation results are poor.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Identify the analytic approach and its epistemology (e.g. deductive content analysis, codebook TA, reflexive TA, IPA).
  2. Classify the task: deductive application of codes, inductive suggestion, summarisation or interpretation.
  3. Assess data sensitivity, consent terms and participant vulnerability, and whether local processing is required.
  4. Assess validation capacity: can humans code a 10-20% sample to measure agreement?
  5. Decide: AI not used, AI as suggester only, or AI as primary coder with validation; document the rationale.
  6. Revisit the decision if agreement falls below the pre-set threshold (e.g. kappa below 0.70) or new risks emerge.

Sources

  • Local Whisper (privacy)
  • data anonymisation pipeline
  • IRR reporting standards
  • publication disclosure norms
  • Braun, V. & Clarke, V. (2022) Thematic Analysis: A Practical Guide
  • Morgan, D. L. (2023) Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT. International Journal of Qualitative Methods
  • The Inquiry Series, C14 — Ethical Constraints & Epistemological Limits of AI-Assisted Qualitative Research

Also known as

  • When AI coding is appropriate vs. inappropriate
  • AI coding fit assessment

Categories

  • Research Operations
  • Research Design & Rigor