Description
Detect and gauge the impact of missing unfavourable studies in a synthesis using funnel plots, asymmetry tests, fail-safe N, trim and fill and selection models.
Publication bias arises when whether a study is published or reported depends on its results; related biases include selective outcome and subgroup reporting and time-lag bias, where null results appear later. Diagnosis uses a funnel plot of effect size against precision or sample size, whose asymmetry suggests small null studies are missing, tested with the Begg-Mazumdar rank correlation or Egger's regression. Impact is gauged with Rosenthal's fail-safe N (the number of unpublished null studies needed to remove significance; above 5k + 10, where k is the number of studies, is conventionally taken as robust), trim and fill, which imputes missing studies and an adjusted pooled effect, selection models with weight functions on p-values or effect sizes, or restriction to large studies. Because the selection mechanism is unknown, adjustments are sensitivity analyses. Asymmetry can also reflect genuine heterogeneity or small-study effects, and funnel plots are informative only with many studies.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Stage
- Strategy
- Data
- Quantitative
- Effort
- Days
Steps
- Draw a funnel plot of effect size against precision (or sample size).
- Test asymmetry with the Begg-Mazumdar rank correlation or Egger's regression of the standard normal deviate on precision.
- Compute fail-safe N and compare it with 5k + 10.
- Apply trim and fill to estimate the number of missing studies and an adjusted pooled effect.
- Fit selection models with p-value or effect-size weight functions.
- Check for outcome-reporting and time-lag biases, and model plausible moderators that could explain asymmetry.
- Report all adjustments as sensitivity analyses alongside the unadjusted estimate.
Sources
- The Handbook of Research Synthesis and Meta-Analysis (Cooper et al., 2009)
- Quantitative Psychological Research (Clark-Carter, 2004)
- Applied Bayesian Modelling (Congdon, 2003)
Also known as
- Funnel plot
- Begg and Mazumdar rank correlation test
- Egger regression test
- Fail-safe N
- File-drawer analysis
- Trim and fill
- Selection models
- Small-study effects
- Outcome reporting bias
- Time-lag bias
- Fail-safe N and funnel graph for the file-drawer problem
- Rosenthal's fail-safe N
- 5k + 10 rule
- Funnel graph
Categories
- Research Operations
- Research Design & Rigor