Description
Estimates a change's effect by comparing the before-after change in a treated group with that in an untreated group, under a parallel-trends assumption.
DiD compares the change over time in outcomes for a treated group with the change for a control group that did not receive treatment. Differencing removes stable unobserved differences between groups and common time trends. The key identifying assumption is parallel trends: absent treatment, both groups would have changed at the same rate, which is assessed with pre-period data and event-study plots. The treatise applies it to product launches and redesigns with a natural control: users in a region without the rollout, or cohorts joining before versus after a change. It separates the change's effect from secular trends and selection differences. Modern staggered-adoption estimators (for example Callaway and Sant'Anna) fix biases of naive two-way fixed effects. Rated HIGH priority.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Goal
- Prove causation
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Identify treated and comparison groups and pre- and post-periods (several pre-periods if possible).
- Plot pre-period trends and test for parallel trends with an event-study specification.
- Estimate the DiD effect with group and time fixed effects and clustered standard errors.
- Use staggered-adoption estimators if rollout timing varies across groups.
- Run placebo tests (fake treatment dates or unaffected outcomes).
- Report the effect with assumptions and robustness checks.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Card D & Krueger AB (1994) Minimum wages and employment. AER 84(4):772-793
- Angrist JD & Pischke J-S (2008) Mostly Harmless Econometrics
- Callaway B & Sant'Anna PHC (2021) Difference-in-differences with multiple time periods. J Econometrics
- Designing Experiments and Analyzing Data (Scott Maxwell & Harold Delaney)
- Methods Atlas (Vivek Gopinath)
- Forward-Thinking Playbook (Vivek Gopinath)
- Mostly Harmless Econometrics (Joshua Angrist & Jörn-Steffen Pischke)
- Causal Inference: The Mixtape (Scott Cunningham)
- Designing Experiments & Analyzing Data (Maxwell/Delaney)
- Methods Atlas (Gopinath)
- Forward-Thinking Playbook (Gopinath)
- The Inquiry Series, Ch 47 — Experimental Design — From A/B to Quasi-Experiments
Also known as
- DiD
- Diff-in-diff
- Controlled before-and-after
- Controlled before-and-after study
- Difference-in-differences
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference