Staged rollout, ramp-up and holdback

Description

Release a change in stages, from insiders and volunteers to random percentages, with pass criteria, guardrails and a lasting holdback, learning safely at each step.

Staged release turns launch into a learning phase. Early rings (team dogfooding, employees, friends and family, alpha and beta users or a pilot site doing real work) catch bugs, misunderstandings, unexpected uses and organisation-system fit problems, with feedback triaged into fixes, support issues and education needs. Opt-in and opt-out releases measure switch-back rates. Randomised ramps then trade speed, quality and risk: under 5% to catch large harms within a day, a maximum power ramp near 50/50 for at least a week, and short operational ramps for peak load, each with pass criteria and near-real-time guardrails, ramping down fast on harm. At full launch a 1 to 5% holdback, or a reverse experiment returning a small group to control, measures lasting effects and validates proxies. When withholding is unacceptable, a randomised phase-in treats late recipients as controls until they receive it.

Attributes

Loss type
Active
Data
Mixed
Effort
Weeks

Steps

  1. Decide in advance what behaviour, metrics and pass criteria each stage must meet, and set up feedback channels that note whether feedback was prompted.
  2. Start with inner rings: team dogfooding, employees, friends and family or a self-contained pilot site doing real work; triage findings into fixes, support issues and education needs.
  3. Optionally release opt-in to volunteers, then opt-out with a clear way back, measuring the switch-back rate.
  4. Ramp randomly: under 5% exposure to catch large harms and platform problems (often within 24 hours), then 10 to 25%, while dialling up automatically against guardrail metrics.
  5. Hold at the maximum power ramp (about 50/50) for at least a week, longer if novelty effects are likely, and analyse segments.
  6. Advance unless results are unsatisfactory, ramp down to zero quickly on harm, and run short operational ramps covering peak load.
  7. At full launch keep a 1 to 5% holdback (or run a reverse experiment) for weeks or months to measure lagging and long-term effects, then release it and clean up code paths.
  8. Where withholding is unacceptable, randomise everyone to early or late groups and compare until the late group is switched on.

Sources

  • UX for Lean Startups (Klein, 2013)
  • Designing for Behavior Change (Wendel, 2014)
  • Designing with Data (King et al., 2017)
  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • Designing Across Senses (Park & Alderman, 2018)
  • Designing the User Interface (Shneiderman et al., 2018)
  • Organizational Implementation (Hertzum, 2021)
  • Handbook of Usability Testing (Rubin & Chisnell, 2008)
  • Product Management for UX People (Crumlish, 2022)
  • Designing Machine Learning Systems (Huyen, 2022)

Also known as

  • Staged feature rollout
  • Partial rollout
  • Percentage rollout
  • Opt-out rate measurement
  • Staggered rollout experiment
  • Randomised phase-in
  • Ramp-up
  • Holdback
  • Phased rollout
  • Controlled exposure
  • SQR ramping framework
  • Long-term holdout
  • Reverse experiment
  • Dogfooding
  • Alpha and beta testing
  • Beta testing
  • Field trial
  • Pilot implementation
  • Staged release learning

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference