Description
Code individual design features of each study in a synthesis and test them as moderators of effect size, instead of applying a single summary quality score.
Study quality is the fit between a study's design and the inferences drawn from it, judged against internal, construct, external and statistical conclusion validity, and is distinct from relevance and from reporting quality. Valentine contrasts a priori exclusion and summary quality scales with post hoc empirical assessment. Jüni and colleagues applied 25 quality scales to 17 trials and found that conclusions flipped depending on the scale, showing such scores are arbitrary. The recommended approach uses few, evidence-based exclusion rules, a separate relevance screen, and coding of individual design components (sampling, assignment mechanism, equating variables and method, sample sizes at assignment and measurement, outcome measure and reliability) with a 'can't tell' option. Each component is then tested empirically as a moderator of effect size. For causal questions, non-randomised studies may be included and compared with randomised ones before any cautious pooling.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Siloed
- Stage
- Strategy
- Data
- Analytical / expert
- Effort
- Weeks
Steps
- Set only highly defensible exclusion criteria backed by evidence for the specific context.
- Screen relevance separately with yes/no items.
- Code design components separately: sampling, assignment mechanism, equating variables and method, sample sizes at assignment and at measurement, outcome measure and score reliability, each with a 'can't tell' option.
- Distinguish what was done from how well it was reported.
- Test each component as a moderator of effect size.
- For causal questions, compare randomised and non-randomised studies before pooling them, and pool only cautiously.
Sources
- The Handbook of Research Synthesis and Meta-Analysis (Cooper et al., 2009)
Also known as
- Judging the quality of primary research
- Risk-of-bias coding
- Quality as a moderator
- Post hoc quality assessment
Categories
- Research Operations
- Research Design & Rigor