Description
Randomises at the level of time periods, regions or groups instead of users when users interfere with each other, avoiding biased estimates in networks and marketplaces.
User-level A/B tests assume no interference (SUTVA). In marketplaces, social networks and shared resources, treating some users changes the experience of others: discounted riders consume drivers' supply, or a feature changes group chats. Cluster randomisation assigns whole groups (cities, teams, communities) to arms; switchback designs alternate treatment and control over time windows within a region. Analysis must account for clustering (intra-cluster correlation drives the design effect) and, for switchbacks, carryover between windows. Rigorous design balances window length against carryover and the number of clusters against power.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Goal
- Prove causation
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Map interference: who affects whom (shared supply, social ties, team accounts).
- Choose the randomisation unit: cluster (region, team, network component) or time window (switchback).
- Estimate the intra-cluster correlation and compute the design effect (1 + (m - 1) x ICC) for sample size.
- For switchbacks, choose window length longer than carryover time and randomise sequences; consider burn-in periods.
- Analyse with cluster-robust or mixed-effects models, or design-based estimators for switchbacks.
- Compare with user-level estimates to quantify interference bias.
Sources
- Hayes & Moulton, Cluster Randomised Trials (2009)
- Bojinov, Simchi-Levi & Zhao, Design and analysis of switchback experiments (2023), Management Science
- Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)
Also known as
- Switchback test
- Time-split experiment
- Cluster randomisation
- Interference-aware experiments
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference