Coding Protocol for Research Synthesis

Description

Build and pilot a coding protocol and manual for extracting study data, train coders, and check reliability, confidence, drift and the sensitivity of results to coding choices.

A synthesis is only as good as its coding of studies. Lipsey sets out what to code: arrays of results plus extrinsic, method and substantive descriptors. Wilson covers explicit eligibility criteria, protocol sections, a relational data structure (study-level and effect-size-level tables linked by study ID), the coding manual, Stock's eight-step coder training, and common mistakes. Orwin and Vevea address sources of coding error and how to reduce them, item-level reliability indices (agreement, kappa, weighted kappa, intercoder correlation, ICC), coder drift detected by randomising coding order and regressing agreement on sequence, confidence ratings that are distinct from reliability, and sensitivity analyses that rerun results under alternative codings of ambiguous items. Coding can take up to eight hours per study, dependent effect sizes must be planned for, and rarely reported items should be dropped.

Attributes

Loss type
Siloed
Stage
Strategy
Effort
Weeks

Steps

  1. Write explicit eligibility criteria and refine them on real studies.
  2. Draft the protocol: report ID, setting, participants, method, treatment, measures, effect-size data and confidence ratings.
  3. Structure data relationally, with study-level and effect-size-level tables linked by study ID.
  4. Pilot the protocol on studies spanning the full range, then revise.
  5. Train coders (for example with Stock's eight steps) until they reach consensus, then double-code and compute item-level reliability.
  6. Randomise coding order and regress per-study agreement on sequence to detect drift.
  7. Rate confidence in each key coded value, and rerun analyses under alternative codings and with and without low-confidence values.

Sources

  • The Handbook of Research Synthesis and Meta-Analysis (Cooper et al., 2009)

Also known as

  • Systematic coding
  • Meta-analytic coding manual
  • Coder training
  • Coder confidence ratings
  • Coder drift check
  • Sensitivity analysis of coding decisions

Categories

  • Research Operations
  • Research Design & Rigor