Description
Replace an experiment's control group with an observational comparison group and test whether observational estimators recover the known experimental effect.
When a randomised experiment exists, its effect is a benchmark against which observational methods can be calibrated. LaLonde took the National Supported Work job-training experiment, swapped its randomised controls for comparison groups drawn from national surveys, and applied the econometric estimators of the day. Their estimates were far off and often had the wrong sign, which pushed policy evaluation towards experiments. Dehejia and Wahba repeated the exercise with propensity scores, checking overlap, trimming controls outside the treated range and comparing covariate balance, and came close to the benchmark. The exercise tells you which observational methods and diagnostics can be trusted in a particular domain. For product teams with both A/B tests and observational analytics, it shows whether quasi-experimental estimates from logs can stand in for experiments.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Obtain a randomised experiment with a credible benchmark effect, such as a past A/B test.
- Build one or more non-experimental comparison groups for the treated units, for example users not exposed in a similar period or market.
- Apply candidate observational estimators (regression adjustment, difference-in-differences, matching, inverse probability weighting).
- Compare each estimate, and its balance and overlap diagnostics, with the experimental benchmark.
- Conclude which methods and diagnostics are trustworthy for similar non-experimental questions in this product, and document the conditions.
Sources
- Causal Inference: The Mixtape (Cunningham, 2021)
Also known as
- Within-study comparison (benchmarking observational estimators against an experiment)
- LaLonde benchmark
- Design replication
- Experimental benchmarking
Categories
- Research Operations
- Research Design & Rigor