Intention-to-Treat Analysis

Description

Analyses all users in the arm they were randomised to, regardless of whether they actually experienced the treatment, preserving the benefits of randomisation.

In product experiments, not everyone assigned to a feature sees it: they never reach the page, the client fails to load, or they opt out. Analysing only those exposed (per-protocol or triggered analysis without care) breaks randomisation if exposure depends on user characteristics affected by treatment. ITT compares groups as randomised, estimating the effect of offering the treatment, which is unbiased though diluted. Triggered analysis is valid only when the trigger condition is identical and computed counterfactually in both arms. Complier average causal effects can then be recovered with IV methods. CONSORT-style flow diagrams make attrition visible.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Record assignment for every randomised user and the point of assignment.
  2. Define exposure precisely and log counterfactual triggers in control (would have seen the feature).
  3. Analyse the primary outcome by assigned arm for all randomised users (ITT).
  4. Run a triggered analysis only on counterfactually defined triggered users in both arms.
  5. Estimate complier effects with IV (assignment as instrument) if needed.
  6. Report a CONSORT-style flow diagram of assignment, exposure and attrition per arm.

Sources

  • Hollis & Campbell, What is meant by intention to treat analysis? (1999), BMJ
  • Schulz, Altman & Moher, CONSORT 2010 Statement (2010), BMJ
  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)

Also known as

  • ITT
  • As-randomised analysis
  • Per-protocol vs ITT

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference