Description
Reconstructs how users actually came to receive a feature or treatment in observational data, so that the assignment can be modelled and adjusted for.
In an RCT the assignment mechanism is known: a coin flip. In product logs, exposure results from user choice, targeting rules, rollout schedules, eligibility logic and algorithms. Rubin argued that causal inference from observational data rests on understanding this mechanism. Reconstruction means documenting the actual rules (feature flags, ranking algorithms, marketing targeting), interviewing engineers and product managers, and modelling the probability of treatment given pre-treatment covariates as a propensity score. Rigour comes from checking overlap (every type of user had a chance of either treatment), covariate balance after weighting or matching, and identifying any deterministic rules that can be exploited as natural experiments.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Goal
- Prove causation
- Stage
- Post-launch
- Data
- Mixed
- Effort
- Days
Steps
- Collect documentation on feature flags, rollout schedules, targeting rules and recommendation logic in force during the study window.
- Interview the engineers and product managers responsible to confirm how exposure was decided.
- Model treatment probability with logistic regression or gradient boosting on pre-treatment covariates to estimate propensity scores.
- Check overlap: trim or flag users with propensity below 0.05 or above 0.95.
- Apply weighting or matching and verify balance with standardised mean differences below 0.1 for all covariates.
- Note any random or arbitrary elements (hash-based rollouts, thresholds) that can be used as instruments or discontinuities.
Sources
- Causal Inference: What If (Miguel Hernán & James Robins)
- DAGs+SEM Playbook (Vivek Gopinath)
- Rosenbaum & Rubin (1983) The central role of the propensity score
- Causal Inference (Hernán & Robins)
- DAGs+SEM Playbook (Gopinath)
- The Inquiry Series, Ch 48 — Target Trial Emulation — Causal Evidence from Log Data
Also known as
- Treatment assignment modelling
- Propensity modelling of exposure
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference