Description
Runs an analysis under every reasonable combination of analytic choices and reports the distribution of results, exposing how fragile a finding is.
Any analysis involves many defensible choices: exclusion rules, metric definitions, time windows, covariates, model types. Each path through this 'garden of forking paths' could give a different answer. Multiverse analysis enumerates the reasonable choices and runs all combinations; specification curve analysis plots the sorted estimates and indicates which choices drive variation, with a permutation-based joint test. If a finding holds across most specifications, it is robust; if it appears only under a few, it was probably a by-product of choice. Product analytics and UX metrics are especially vulnerable because metric definitions are flexible.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- List every analytic decision point and the reasonable options at each (e.g. 3 exclusion rules x 4 windows x 2 models).
- Justify inclusion of options in advance, ideally pre-registered.
- Run the analysis for every combination and store estimates and uncertainty.
- Plot the specification curve: estimates sorted by size with indicators of which choices were used.
- Conduct inference across specifications (e.g. permutation test of the share of significant same-sign results).
- Report the full distribution and identify choices that change conclusions.
Sources
- Steegen et al., Increasing transparency through a multiverse analysis (2016), Perspectives on Psychological Science
- Simonsohn, Simmons & Nelson, Specification curve analysis (2020), Nature Human Behaviour
Also known as
- Multiverse analysis
- Specification curve
- Robustness to analytic choices
Categories
- Research Operations
- Research Design & Rigor