Difference-in-Differences (DiD)

Description

Estimates a change's effect by comparing the before-after change in a treated group with that in an untreated group, under a parallel-trends assumption.

DiD compares the change over time in outcomes for a treated group with the change for a control group that did not receive treatment. Differencing removes stable unobserved differences between groups and common time trends. The key identifying assumption is parallel trends: absent treatment, both groups would have changed at the same rate, which is assessed with pre-period data and event-study plots. The treatise applies it to product launches and redesigns with a natural control: users in a region without the rollout, or cohorts joining before versus after a change. It separates the change's effect from secular trends and selection differences. Modern staggered-adoption estimators (for example Callaway and Sant'Anna) fix biases of naive two-way fixed effects. Rated HIGH priority.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Identify treated and comparison groups and pre- and post-periods (several pre-periods if possible).
  2. Plot pre-period trends and test for parallel trends with an event-study specification.
  3. Estimate the DiD effect with group and time fixed effects and clustered standard errors.
  4. Use staggered-adoption estimators if rollout timing varies across groups.
  5. Run placebo tests (fake treatment dates or unaffected outcomes).
  6. Report the effect with assumptions and robustness checks.

Sources

  • Lost Methods in UX — A Complete Treatise (Gopinath)
  • Card D & Krueger AB (1994) Minimum wages and employment. AER 84(4):772-793
  • Angrist JD & Pischke J-S (2008) Mostly Harmless Econometrics
  • Callaway B & Sant'Anna PHC (2021) Difference-in-differences with multiple time periods. J Econometrics
  • Designing Experiments and Analyzing Data (Scott Maxwell & Harold Delaney)
  • Methods Atlas (Vivek Gopinath)
  • Forward-Thinking Playbook (Vivek Gopinath)
  • Mostly Harmless Econometrics (Joshua Angrist & Jörn-Steffen Pischke)
  • Causal Inference: The Mixtape (Scott Cunningham)
  • Designing Experiments & Analyzing Data (Maxwell/Delaney)
  • Methods Atlas (Gopinath)
  • Forward-Thinking Playbook (Gopinath)
  • The Inquiry Series, Ch 47 — Experimental Design — From A/B to Quasi-Experiments

Also known as

  • DiD
  • Diff-in-diff
  • Controlled before-and-after
  • Controlled before-and-after study
  • Difference-in-differences

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference