Multiverse and Specification Curve Analysis

Description

Runs an analysis under every reasonable combination of analytic choices and reports the distribution of results, exposing how fragile a finding is.

Any analysis involves many defensible choices: exclusion rules, metric definitions, time windows, covariates, model types. Each path through this 'garden of forking paths' could give a different answer. Multiverse analysis enumerates the reasonable choices and runs all combinations; specification curve analysis plots the sorted estimates and indicates which choices drive variation, with a permutation-based joint test. If a finding holds across most specifications, it is robust; if it appears only under a few, it was probably a by-product of choice. Product analytics and UX metrics are especially vulnerable because metric definitions are flexible.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. List every analytic decision point and the reasonable options at each (e.g. 3 exclusion rules x 4 windows x 2 models).
  2. Justify inclusion of options in advance, ideally pre-registered.
  3. Run the analysis for every combination and store estimates and uncertainty.
  4. Plot the specification curve: estimates sorted by size with indicators of which choices were used.
  5. Conduct inference across specifications (e.g. permutation test of the share of significant same-sign results).
  6. Report the full distribution and identify choices that change conclusions.

Sources

  • Steegen et al., Increasing transparency through a multiverse analysis (2016), Perspectives on Psychological Science
  • Simonsohn, Simmons & Nelson, Specification curve analysis (2020), Nature Human Behaviour

Also known as

  • Multiverse analysis
  • Specification curve
  • Robustness to analytic choices

Categories

  • Research Operations
  • Research Design & Rigor