Description
Release a change in stages, from insiders and volunteers to random percentages, with pass criteria, guardrails and a lasting holdback, learning safely at each step.
Staged release turns launch into a learning phase. Early rings (team dogfooding, employees, friends and family, alpha and beta users or a pilot site doing real work) catch bugs, misunderstandings, unexpected uses and organisation-system fit problems, with feedback triaged into fixes, support issues and education needs. Opt-in and opt-out releases measure switch-back rates. Randomised ramps then trade speed, quality and risk: under 5% to catch large harms within a day, a maximum power ramp near 50/50 for at least a week, and short operational ramps for peak load, each with pass criteria and near-real-time guardrails, ramping down fast on harm. At full launch a 1 to 5% holdback, or a reverse experiment returning a small group to control, measures lasting effects and validates proxies. When withholding is unacceptable, a randomised phase-in treats late recipients as controls until they receive it.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Evaluation, Post-launch
- Data
- Mixed
- Effort
- Weeks
Steps
- Decide in advance what behaviour, metrics and pass criteria each stage must meet, and set up feedback channels that note whether feedback was prompted.
- Start with inner rings: team dogfooding, employees, friends and family or a self-contained pilot site doing real work; triage findings into fixes, support issues and education needs.
- Optionally release opt-in to volunteers, then opt-out with a clear way back, measuring the switch-back rate.
- Ramp randomly: under 5% exposure to catch large harms and platform problems (often within 24 hours), then 10 to 25%, while dialling up automatically against guardrail metrics.
- Hold at the maximum power ramp (about 50/50) for at least a week, longer if novelty effects are likely, and analyse segments.
- Advance unless results are unsatisfactory, ramp down to zero quickly on harm, and run short operational ramps covering peak load.
- At full launch keep a 1 to 5% holdback (or run a reverse experiment) for weeks or months to measure lagging and long-term effects, then release it and clean up code paths.
- Where withholding is unacceptable, randomise everyone to early or late groups and compare until the late group is switched on.
Sources
- UX for Lean Startups (Klein, 2013)
- Designing for Behavior Change (Wendel, 2014)
- Designing with Data (King et al., 2017)
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
- Designing Across Senses (Park & Alderman, 2018)
- Designing the User Interface (Shneiderman et al., 2018)
- Organizational Implementation (Hertzum, 2021)
- Handbook of Usability Testing (Rubin & Chisnell, 2008)
- Product Management for UX People (Crumlish, 2022)
- Designing Machine Learning Systems (Huyen, 2022)
Also known as
- Staged feature rollout
- Partial rollout
- Percentage rollout
- Opt-out rate measurement
- Staggered rollout experiment
- Randomised phase-in
- Ramp-up
- Holdback
- Phased rollout
- Controlled exposure
- SQR ramping framework
- Long-term holdout
- Reverse experiment
- Dogfooding
- Alpha and beta testing
- Beta testing
- Field trial
- Pilot implementation
- Staged release learning
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference