Description
Estimates a causal effect by comparing users just above and just below a sharp eligibility threshold that determines treatment.
When treatment is assigned by a known cutoff on a running variable (score, session count, date), units just either side of the cutoff should be comparable on everything but treatment. Fitting local regressions on each side and measuring the jump at the threshold gives a local average treatment effect for users near the cutoff. The treatise's examples: users with more than X sessions get a new onboarding, users scoring above Y are promoted to a premium tier, users who joined before date Z kept the old interface. Validity requires that users cannot precisely manipulate the running variable (checked with a McCrary density test) and that covariates are balanced at the cutoff. Fuzzy RDD handles imperfect compliance.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Goal
- Prove causation
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Identify a rule assigning treatment at a known threshold of a running variable.
- Plot outcomes against the running variable and inspect for a discontinuity.
- Test for manipulation with a density test and check covariate balance at the cutoff.
- Estimate the jump with local linear regression using a data-driven bandwidth.
- Check robustness to bandwidth, polynomial order and placebo cutoffs.
- Report the local effect and its limited generalisability away from the cutoff.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Thistlethwaite DL & Campbell DT (1960) Regression-discontinuity analysis. J Educ Psychol 51(6):309-317
- Lee DS & Lemieux T (2010) Regression discontinuity designs in economics. J Econ Lit 48(2)
- Cattaneo MD, Idrobo N & Titiunik R (2019) A Practical Introduction to Regression Discontinuity Designs
- Designing Experiments and Analyzing Data (Scott Maxwell & Harold Delaney)
- Methods Atlas (Vivek Gopinath)
- Forward-Thinking Playbook (Vivek Gopinath)
- Thistlethwaite & Campbell (1960) Regression-discontinuity analysis
- A Practical Introduction to Regression Discontinuity Designs (Cattaneo, Idrobo & Titiunik)
- Designing Experiments & Analyzing Data (Maxwell/Delaney)
- Methods Atlas (Gopinath)
- Forward-Thinking Playbook (Gopinath)
- The Inquiry Series, Ch 47 — Experimental Design — From A/B to Quasi-Experiments
Also known as
- RDD
- Regression discontinuity
- Regression discontinuity design
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference