Meta-Analysis of UX Studies

Description

Statistically combines effect sizes from multiple studies or experiments, estimating an overall effect, its heterogeneity and moderators.

Organisations run hundreds of studies and experiments but rarely combine them. Meta-analysis converts each study's result into a common effect size with its variance, pools them (fixed or random effects, the latter allowing true effects to vary), and quantifies heterogeneity (tau squared, I squared, prediction intervals). Meta-regression explores moderators like platform or market. Publication bias checks (funnel plots, Egger's test) are directly relevant to internal research too, as unreported null tests bias institutional memory. Within a company, meta-analysis of past A/B tests yields realistic priors and shows whether design patterns generalise.

Attributes

Loss type
Latent
Stage
Strategy
Effort
Weeks

Steps

  1. Define the question and inclusion criteria for studies or experiments.
  2. Extract effect sizes and their standard errors consistently (e.g. log odds ratios, standardised mean differences).
  3. Fit a random-effects model and report the pooled effect, confidence interval and prediction interval.
  4. Quantify heterogeneity (I squared, tau) and run meta-regression on planned moderators.
  5. Assess small-study or publication bias with funnel plots and sensitivity analyses.
  6. Summarise implications, including when the pattern should not be expected to replicate.

Sources

  • Borenstein, Hedges, Higgins & Rothstein, Introduction to Meta-Analysis (2009)
  • DerSimonian & Laird, Meta-analysis in clinical trials (1986), Controlled Clinical Trials
  • Higgins et al. (eds.), Cochrane Handbook for Systematic Reviews of Interventions (2019)

Also known as

  • Meta-analysis
  • Random-effects meta-analysis
  • Research synthesis (quantitative)

Categories

  • Research Operations
  • Research Design & Rigor