Description
Chance-corrected agreement statistics showing qualitative coding is systematic rather than one researcher's opinion: κ for two coders, α for more coders, missing data or ordinal scales.
When coding qualitative data such as error types, coders may disagree; percent agreement overstates reliability because some agreement occurs by chance. Cohen's κ = (Po − Pe)/(1 − Pe), where Po is observed agreement and Pe chance-expected agreement computed from row and column proportions. Landis and Koch bands: < 0.20 poor, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Krippendorff's α = 1 − Do/De generalises to any number of coders, missing data and nominal, ordinal or interval data; α ≥ 0.80 is reliable, 0.67–0.80 tentatively acceptable. Vivek Gopinath's playbooks require double-coding 30–40% of sessions and κ ≥ 0.70, with codebook refinement and re-coding (including a 20% re-code sample) if lower. In his worked example, two coders classify 60 breakdowns: Po = 0.833, Pe = 0.366, κ = 0.737, substantial agreement.
Attributes
- Discipline
- Qualitative Analysis & Synthesis
- Loss type
- Vestigial
- Stage
- Definition
- Data
- Qualitative
- Effort
- Hours
Steps
- Finalise a codebook with definitions and examples before coding.
- Have two or more coders independently code 30–40% of sessions.
- Build a coder A × coder B contingency table.
- Compute Po as diagonal agreements / total and Pe as the sum over categories of row proportion × column proportion.
- Compute κ = (Po − Pe)/(1 − Pe), or Krippendorff's α for more than two coders, missing data or ordinal data.
- If κ < 0.70, refine the codebook, retrain and re-code; report the statistic alongside findings.
Sources
- Vivek Gopinath (2025), Tree Testing and Error Taxonomy: A Complete Chapter for UX Researchers
- Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation
- Vivek Gopinath (2025), Detecting Behavioral Drift: A Mixed Methods Playbook and E-commerce Case Study
- Sage Handbook of Qualitative Data Analysis (Flick)
- Research Methods in HCI (Lazar)
- Statistics for HCI (Dix)
- Rapid Contextual Design (Morgan Kaufmann)
- Error Taxonomy Playbook (Gopinath)
- Anthropic Claude API (claude-sonnet-4-20250514)
- structured outputs (JSON schema)
- asyncio
- A Coefficient of Agreement for Nominal Scales (Cohen)
- Measuring Nominal Scale Agreement Among Many Raters (Fleiss)
- The Inquiry Series, Ch 34 — Framework Analysis & Systematic Qualitative Synthesis
- The Inquiry Series, C12 — AI Agents for Structured Qualitative Analysis at Scale
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
Also known as
- Inter-rater reliability
- IRR
- Inter-coder agreement
- Intercoder reliability
- Cohen's kappa
- Fleiss' kappa
- Coding agreement
- Inter-Rater Reliability (Cohen's and Fleiss' Kappa)
Categories
- User Research
- Qualitative Analysis & Synthesis