Description
Uses a variable that shifts exposure but affects outcomes only through exposure to estimate causal effects when users self-select into treatment.
An instrument must be relevant (it strongly predicts treatment) and valid (it affects the outcome only through treatment, the exclusion restriction). Two-stage least squares estimates the effect for compliers, the local average treatment effect. The treatise's transfer: when you cannot randomise who uses a new UI, find a plausible instrument, such as a server-side flag randomly assigned for technical reasons, a geographic boundary creating differential access, or a temporal event creating quasi-random exposure. Encouragement designs, where users are randomly nudged toward a feature, are the most practical UX instrument. Weak instruments (first-stage F below about 10) give biased estimates, and exclusion must be argued, not tested.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Goal
- Prove causation
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Define the treatment (actual exposure) and the outcome; identify a candidate instrument.
- Argue relevance and exclusion explicitly, ideally with a DAG.
- Estimate the first stage and check instrument strength (F above 10).
- Estimate the effect with 2SLS and robust standard errors.
- Characterise compliers and state that the estimate is a LATE.
- Test sensitivity to plausible exclusion violations.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Angrist JD, Imbens GW & Rubin DB (1996) Identification of causal effects using instrumental variables. JASA 91:444-455
- Angrist JD & Pischke J-S (2008) Mostly Harmless Econometrics
- Angrist, Imbens & Rubin, Identification of causal effects using instrumental variables (1996), JASA
- Angrist & Pischke, Mostly Harmless Econometrics (2009)
- Cinelli, C., Forney, A. & Pearl, J., A Crash Course in Good and Bad Controls (2022)
Also known as
- IV estimation
- Two-stage least squares (2SLS)
- Encouragement design analysis
- IV
- Two-stage least squares
- 2SLS
- Instrumental Variables Analysis
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference