Multi-Country Collaborative Codebook Calibration

Description

Calibrate one code system across teams coding in different languages through shared test-coding rounds, then code locally with counter-checking, selective translation and a common report structure.

When several teams analyse interviews in their own languages, a shared code list alone does not guarantee they apply it the same way. In the EU-funded BeLL study, ten national teams coded 82 interviews with one system built from theory-based categories and earlier survey codes. Two test-coding rounds calibrated it: every team coded the same English interview, a small working group collated disagreements and proposed new codes for all teams to discuss, the list was revised, and the process was repeated with a second shared interview plus national ones. Afterwards only minor code changes were allowed, and teams could add country-only codes. Full coding used a main coder with a second researcher counter-checking every coding, a shared forum for questions, memos on cultural differences, translation of about a quarter of the material into English for joint checks, and national reports in a common structure.

Attributes

Loss type
Siloed
Effort
Months

Steps

  1. Build a start list from theory-defined categories and earlier codes, with a glossary and versioned guidelines.
  2. Test round 1: all teams code the same shared transcript; a working group collates disagreements and proposed codes and puts them to all teams; revise the list.
  3. Test round 2: all teams code a second shared transcript plus one or two local-language transcripts; repeat collation and revision until only minor changes remain.
  4. Allow local-only codes for country specifics and keep them out of cross-national comparisons.
  5. Full coding: each team codes its own data, with a second researcher counter-checking every coding and unresolved issues discussed across teams using translated passages.
  6. Translate a defined share of material (about 25%) by set criteria for shared analysis and quality checks, and keep memos and a shared forum log throughout.
  7. Have each team write a national report in a common structure, then synthesise an overall report with partner feedback.

Sources

  • BeLL Qualitative Data Analysis Report (Sgier, 2014)

Also known as

  • Distributed multilingual team coding
  • Cross-national qualitative coding with test-coding rounds
  • Collaborative code system calibration
  • Multi-country collaborative codebook calibration and coding

Categories

  • User Research
  • Qualitative Analysis & Synthesis