Vote-Counting Procedures

Description

Estimate a pooled effect from the direction or significance of study results when effect sizes are unavailable, and combine it with effect sizes from fully reported studies.

Conventional vote counting tallies studies as significant positive, significant negative or null and declares the modal category the winner. It ignores sample size, loses power as studies accumulate, and can conclude 'no effect' when a real effect exists. The principled alternatives are the sign test (a binomial test with p = .5 on the direction of results), maximum-likelihood vote-count estimators that recover an effect size d or r and a confidence interval from the counts and sample sizes, and Bushman and Wang's combined procedure that pools vote-count estimates with effect sizes from fully reported studies. They should be used only for studies lacking the statistics needed for an effect size. They assume a fixed common effect, give no information about heterogeneity, and direction-only estimators are at best 64% as efficient as effect-size methods.

Attributes

Loss type
Siloed
Stage
Strategy
Effort
Days

Steps

  1. Identify the studies that report only direction or significance and cannot yield an effect size.
  2. Classify each as positive or negative in direction, or as significant positive versus not.
  3. Apply the sign test (binomial with p = .5) to the direction counts.
  4. Estimate the common effect by maximum likelihood from the counts and sample sizes, and form a confidence interval.
  5. Pool the vote-count estimate with effect-size estimates from fully reported studies using the combined procedure.
  6. Report the fixed-effect assumption and the efficiency loss alongside the result.

Sources

  • The Handbook of Research Synthesis and Meta-Analysis (Cooper et al., 2009)

Also known as

  • Vote counting
  • Sign test for study outcomes
  • Vote-count confidence interval
  • Combined vote-count and effect-size procedure

Categories

  • Analytics & Behaviour
  • Statistics & Modelling