Publication Bias Diagnosis and Adjustment

Description

Detect and gauge the impact of missing unfavourable studies in a synthesis using funnel plots, asymmetry tests, fail-safe N, trim and fill and selection models.

Publication bias arises when whether a study is published or reported depends on its results; related biases include selective outcome and subgroup reporting and time-lag bias, where null results appear later. Diagnosis uses a funnel plot of effect size against precision or sample size, whose asymmetry suggests small null studies are missing, tested with the Begg-Mazumdar rank correlation or Egger's regression. Impact is gauged with Rosenthal's fail-safe N (the number of unpublished null studies needed to remove significance; above 5k + 10, where k is the number of studies, is conventionally taken as robust), trim and fill, which imputes missing studies and an adjusted pooled effect, selection models with weight functions on p-values or effect sizes, or restriction to large studies. Because the selection mechanism is unknown, adjustments are sensitivity analyses. Asymmetry can also reflect genuine heterogeneity or small-study effects, and funnel plots are informative only with many studies.

Attributes

Loss type
Unmigrated
Stage
Strategy
Effort
Days

Steps

  1. Draw a funnel plot of effect size against precision (or sample size).
  2. Test asymmetry with the Begg-Mazumdar rank correlation or Egger's regression of the standard normal deviate on precision.
  3. Compute fail-safe N and compare it with 5k + 10.
  4. Apply trim and fill to estimate the number of missing studies and an adjusted pooled effect.
  5. Fit selection models with p-value or effect-size weight functions.
  6. Check for outcome-reporting and time-lag biases, and model plausible moderators that could explain asymmetry.
  7. Report all adjustments as sensitivity analyses alongside the unadjusted estimate.

Sources

  • The Handbook of Research Synthesis and Meta-Analysis (Cooper et al., 2009)
  • Quantitative Psychological Research (Clark-Carter, 2004)
  • Applied Bayesian Modelling (Congdon, 2003)

Also known as

  • Funnel plot
  • Begg and Mazumdar rank correlation test
  • Egger regression test
  • Fail-safe N
  • File-drawer analysis
  • Trim and fill
  • Selection models
  • Small-study effects
  • Outcome reporting bias
  • Time-lag bias
  • Fail-safe N and funnel graph for the file-drawer problem
  • Rosenthal's fail-safe N
  • 5k + 10 rule
  • Funnel graph

Categories

  • Research Operations
  • Research Design & Rigor