Description
Choose whether to randomise by page, session, user, device, account or cluster by weighing experience consistency, which metrics must be valid, and interference between units.
The unit an experiment randomises determines what it can measure. Finer units (pageview, session) add statistical power but break when a feature or metric spans units, or when users notice an inconsistent experience; coarser units (user, account, tenant, cluster) keep experiences consistent and support user-level and long-term metrics. The randomisation unit should be equal to or coarser than the analysis unit; when coarser, variance must use the delta method or bootstrap and per-user contributions may need capping. Signed-in IDs suit cross-device and long-term work; cookies or device IDs suit flows that cross sign-in; IP addresses are reserved for infrastructure tests because their sizes vary wildly. In enterprise products, advertising and social networks, randomising tenants, advertisers or friend clusters may be needed to avoid interference.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Definition, Evaluation
- Data
- Analytical / expert
- Effort
- Hours
Steps
- List candidate units: pageview, session, device or cookie, signed-in user, account or tenant, cluster.
- Ask whether users would notice inconsistency if their experience changed between units; if so, move to a coarser unit.
- List the metrics that must be valid (for example sessions per user needs at least user-level randomisation) and match the unit to them.
- Check for carryover between units and interference through shared accounts, marketplaces or social ties; consider cluster randomisation if present.
- Choose the ID type by scope and longevity (signed-in ID for cross-device and long-term, cookie for pre-sign-in flows, IP only for infrastructure).
- Plan the variance method that matches the unit (delta method or bootstrap when the analysis unit is finer than the randomisation unit).
Sources
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
- Design and Analysis of Experiments, Vol. 1 (Hinkelmann & Kempthorne, 2008)
- Beyond t Test and ANOVA (Yu et al., 2022)
Also known as
- Choosing a randomisation unit
- Randomization unit selection
- Unit of diversion
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference