Switchback and Cluster-Randomised Experiments

Description

Randomises at the level of time periods, regions or groups instead of users when users interfere with each other, avoiding biased estimates in networks and marketplaces.

User-level A/B tests assume no interference (SUTVA). In marketplaces, social networks and shared resources, treating some users changes the experience of others: discounted riders consume drivers' supply, or a feature changes group chats. Cluster randomisation assigns whole groups (cities, teams, communities) to arms; switchback designs alternate treatment and control over time windows within a region. Analysis must account for clustering (intra-cluster correlation drives the design effect) and, for switchbacks, carryover between windows. Rigorous design balances window length against carryover and the number of clusters against power.

Attributes

Loss type
Latent
Stage
Post-launch
Effort
Weeks

Steps

  1. Map interference: who affects whom (shared supply, social ties, team accounts).
  2. Choose the randomisation unit: cluster (region, team, network component) or time window (switchback).
  3. Estimate the intra-cluster correlation and compute the design effect (1 + (m - 1) x ICC) for sample size.
  4. For switchbacks, choose window length longer than carryover time and randomise sequences; consider burn-in periods.
  5. Analyse with cluster-robust or mixed-effects models, or design-based estimators for switchbacks.
  6. Compare with user-level estimates to quantify interference bias.

Sources

  • Hayes & Moulton, Cluster Randomised Trials (2009)
  • Bojinov, Simchi-Levi & Zhao, Design and analysis of switchback experiments (2023), Management Science
  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020)

Also known as

  • Switchback test
  • Time-split experiment
  • Cluster randomisation
  • Interference-aware experiments

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference