Description
Before optimising something, deliberately remove a feature or degrade a variable such as speed in a controlled test to learn how much it matters.
Improving a feature or making a system faster is costly; removing or slowing it is cheap. 'Experiment 0' asks what experiment should come before the planned one. Removing an existing feature shows whether it has value (Skyscanner deleted a flight ticker that moved no metric; Static Pixels removed a pre-authorisation step and transactions doubled). Latency injection adds a constant delay to treatment to estimate what a speed-up would be worth: assume local linearity and verify it with at least two delays (Bing used 100 and 250 ms; the larger gave about 2.5 times the effect), insert the delay where it mimics server slowness, and delay specific regions to learn where speed matters. The HCI tradition varies system response time in the lab, measuring task time, errors, strategy shifts, stress and satisfaction to find an appropriate pace for a task and user group.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Definition, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Before the first planned optimisation experiment, ask which experiment should precede it: removal, painted door or degradation.
- For removal, instrument the feature, remove it for a randomised group and keep measuring engagement and the key metric; interpret protests from existing users and requests from new cohorts.
- For latency, choose delay points that mimic real server slowness and run at least two delay levels (for example 100 and 250 ms) to check linearity.
- Measure the overall evaluation criterion plus perceived-performance metrics (time to first result, above-the-fold time, Speed Index, time to successful click).
- In lab studies, define response time and think time, randomise or counterbalance delay conditions, and measure task time, errors, strategy change and satisfaction for representative tasks.
- Estimate value per millisecond or per feature, extrapolate cautiously to plausible speed-ups, and decide whether to invest, delete or deprioritise.
Sources
- Designing with Data (King et al., 2017)
- Designing the User Interface (Shneiderman et al., 2018)
- Lean Analytics (Croll & Yoskovitz, 2013)
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
Also known as
- Experiment 0
- Feature removal test
- MVP feature kill test
- Latency injection test
- Slowdown experiment
- Speed experiment
- System response time (latency) study
- SRT experiment
- Delay tolerance study
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference