Instrumental Variables (IV)

Description

Uses a variable that shifts exposure but affects outcomes only through exposure to estimate causal effects when users self-select into treatment.

An instrument must be relevant (it strongly predicts treatment) and valid (it affects the outcome only through treatment, the exclusion restriction). Two-stage least squares estimates the effect for compliers, the local average treatment effect. The treatise's transfer: when you cannot randomise who uses a new UI, find a plausible instrument, such as a server-side flag randomly assigned for technical reasons, a geographic boundary creating differential access, or a temporal event creating quasi-random exposure. Encouragement designs, where users are randomly nudged toward a feature, are the most practical UX instrument. Weak instruments (first-stage F below about 10) give biased estimates, and exclusion must be argued, not tested.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Define the treatment (actual exposure) and the outcome; identify a candidate instrument.
  2. Argue relevance and exclusion explicitly, ideally with a DAG.
  3. Estimate the first stage and check instrument strength (F above 10).
  4. Estimate the effect with 2SLS and robust standard errors.
  5. Characterise compliers and state that the estimate is a LATE.
  6. Test sensitivity to plausible exclusion violations.

Sources

  • Lost Methods in UX — A Complete Treatise (Gopinath)
  • Angrist JD, Imbens GW & Rubin DB (1996) Identification of causal effects using instrumental variables. JASA 91:444-455
  • Angrist JD & Pischke J-S (2008) Mostly Harmless Econometrics
  • Angrist, Imbens & Rubin, Identification of causal effects using instrumental variables (1996), JASA
  • Angrist & Pischke, Mostly Harmless Econometrics (2009)
  • Cinelli, C., Forney, A. & Pearl, J., A Crash Course in Good and Bad Controls (2022)

Also known as

  • IV estimation
  • Two-stage least squares (2SLS)
  • Encouragement design analysis
  • IV
  • Two-stage least squares
  • 2SLS
  • Instrumental Variables Analysis

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference