Description
Records hypotheses, primary outcomes, sample size and analysis plan before data collection to prevent outcome switching and p-hacking.
The treatise repeatedly cites pre-registration: research protocol pre-registration as a DAG revival context, and pre-registration of hypotheses as missing from industry A/B testing. Pre-registration time-stamps a plan specifying hypotheses, primary and secondary outcomes, sample size with power justification, exclusion rules and analysis, separating confirmatory from exploratory findings. It counters the garden of forking paths, where flexible analysis turns noise into apparent effects. In product organisations it can be lightweight: an internal registry where experiment and study plans are frozen before launch and results are reported against them, including null results.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Goal
- Support a decision
- Stage
- Definition, Evaluation
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Write hypotheses and designate one or a few primary outcomes.
- Justify sample size with a power or precision analysis.
- Specify exclusion rules, analysis models, multiplicity corrections and stopping rules.
- Register the plan in an internal or public registry with a timestamp.
- Run the study and analyse as planned; label deviations and exploratory analyses.
- Report all registered outcomes, including nulls.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Nosek BA et al. (2018) The preregistration revolution. PNAS 115(11)
- Simmons JP, Nelson LD & Simonsohn U (2011) False-positive psychology. Psychological Science
- Statistics for Dummies series (Rumsey)
- Designing Experiments & Analyzing Data (Maxwell & Delaney)
- The Inquiry Series, Ch 41 — Frequentist Inference Done Right
Also known as
- Study pre-registration
- Pre-specified analysis plan
- Analysis pre-registration
- Pre-analysis plan
- Registered report
- Pre-registration of UX Studies
Categories
- Research Operations
- Research Design & Rigor