Causal effect estimation from observational data

Description

Estimates the effect of a feature or intervention from logs or surveys by adjusting for confounding with methods such as weighting, matching, g-computation or doubly robust estimation.

When experiments are impossible, causal effects can still be estimated under the assumptions of exchangeability (no unmeasured confounding given covariates), positivity (every user could have received each treatment) and consistency (a well-defined treatment). Estimators include outcome regression (g-computation), inverse probability of treatment weighting, matching, and doubly robust methods such as augmented IPW or targeted maximum likelihood, which remain consistent if either the treatment or outcome model is correct. For time-varying treatments, marginal structural models handle confounders affected by earlier treatment. Rigour requires a DAG-justified adjustment set, balance and overlap diagnostics, bootstrapped intervals and sensitivity analysis such as E-values for unmeasured confounding.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Weeks

Steps

  1. Specify the causal question and target trial, then select covariates from a DAG.
  2. Estimate propensity scores and an outcome model; inspect overlap and trim extreme weights (e.g. truncate at the 1st and 99th percentiles).
  3. Estimate the effect with a doubly robust estimator (AIPW or TMLE) and compare with IPW and g-computation estimates.
  4. Verify covariate balance after weighting with standardised mean differences below 0.1.
  5. Compute bootstrap confidence intervals with at least 1,000 resamples.
  6. Report an E-value or similar sensitivity analysis showing how strong unmeasured confounding would need to be to explain away the effect.

Sources

  • Causal Inference: What If (Miguel Hernán & James Robins)
  • DAGs+SEM Playbook (Vivek Gopinath)
  • VanderWeele & Ding (2017) Sensitivity analysis in observational research: introducing the E-value
  • Causal Inference (Hernán & Robins)
  • DAGs+SEM Playbook (Gopinath)
  • The Inquiry Series, Ch 48 — Target Trial Emulation — Causal Evidence from Log Data

Also known as

  • Observational causal inference
  • G-methods
  • Inverse probability weighting
  • Propensity score methods

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference