Cohen's Kappa and Krippendorff's Alpha (Inter-Rater Reliability)

Description

Chance-corrected agreement statistics showing qualitative coding is systematic rather than one researcher's opinion: κ for two coders, α for more coders, missing data or ordinal scales.

When coding qualitative data such as error types, coders may disagree; percent agreement overstates reliability because some agreement occurs by chance. Cohen's κ = (Po − Pe)/(1 − Pe), where Po is observed agreement and Pe chance-expected agreement computed from row and column proportions. Landis and Koch bands: < 0.20 poor, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Krippendorff's α = 1 − Do/De generalises to any number of coders, missing data and nominal, ordinal or interval data; α ≥ 0.80 is reliable, 0.67–0.80 tentatively acceptable. Vivek Gopinath's playbooks require double-coding 30–40% of sessions and κ ≥ 0.70, with codebook refinement and re-coding (including a 20% re-code sample) if lower. In his worked example, two coders classify 60 breakdowns: Po = 0.833, Pe = 0.366, κ = 0.737, substantial agreement.

Attributes

Loss type
Vestigial
Stage
Definition
Effort
Hours

Steps

  1. Finalise a codebook with definitions and examples before coding.
  2. Have two or more coders independently code 30–40% of sessions.
  3. Build a coder A × coder B contingency table.
  4. Compute Po as diagonal agreements / total and Pe as the sum over categories of row proportion × column proportion.
  5. Compute κ = (Po − Pe)/(1 − Pe), or Krippendorff's α for more than two coders, missing data or ordinal data.
  6. If κ < 0.70, refine the codebook, retrain and re-code; report the statistic alongside findings.

Sources

  • Vivek Gopinath (2025), Tree Testing and Error Taxonomy: A Complete Chapter for UX Researchers
  • Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation
  • Vivek Gopinath (2025), Detecting Behavioral Drift: A Mixed Methods Playbook and E-commerce Case Study
  • Sage Handbook of Qualitative Data Analysis (Flick)
  • Research Methods in HCI (Lazar)
  • Statistics for HCI (Dix)
  • Rapid Contextual Design (Morgan Kaufmann)
  • Error Taxonomy Playbook (Gopinath)
  • Anthropic Claude API (claude-sonnet-4-20250514)
  • structured outputs (JSON schema)
  • asyncio
  • A Coefficient of Agreement for Nominal Scales (Cohen)
  • Measuring Nominal Scale Agreement Among Many Raters (Fleiss)
  • The Inquiry Series, Ch 34 — Framework Analysis & Systematic Qualitative Synthesis
  • The Inquiry Series, C12 — AI Agents for Structured Qualitative Analysis at Scale
  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)

Also known as

  • Inter-rater reliability
  • IRR
  • Inter-coder agreement
  • Intercoder reliability
  • Cohen's kappa
  • Fleiss' kappa
  • Coding agreement
  • Inter-Rater Reliability (Cohen's and Fleiss' Kappa)

Categories

  • User Research
  • Qualitative Analysis & Synthesis