Description
Check whether an experiment's effect decays (novelty) or grows (learning, change aversion) over time, and use cohort, post-period or staggered designs to estimate the long-term effect.
Short A/B tests assume a constant treatment effect, but visible changes often provoke novelty clicks or change aversion that fade as users adapt. Per-day effect plots, using each day's own visitors rather than cumulative plots that create false trends, show decline (novelty) or growth (primacy, learning). Segmenting by usage frequency helps: in Google's 2010 search redesign, high-frequency users who met the change most looked better early, and others converged after a longer run; large visual changes may need about six weeks. New users are a check: if a change wins only for them, suspect novelty. For durable estimates, cohort analysis tracks stable signed-in users; post-period analysis switches the treatment off and measures the residual learned effect (Google found users learned to click fewer ads as ad load rose); time-staggered treatments start a second copy later and declare convergence when the two agree.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Plot the treatment effect for each day using that day's visitors only, and plot the effect for the cohort first exposed on day one or two.
- Segment by usage frequency and tenure: if high-frequency users already look better, exposure and learning are driving the early result.
- Compare new users: a change that is flat or negative for them is not explained by habituation; one that wins only for them suggests novelty.
- If a trend appears, extend the run until the effect stabilises (large visual changes may need about six weeks) and make longer durations the default for visual changes.
- When the long-term effect matters, choose a design: stable signed-in cohort reweighted by pre-experiment strata, post-period analysis after switching the treatment off, or a time-staggered second treatment copy.
- Test convergence, separate user-learned from system-learned effects, and model learned effects to forecast long-term impact from future short tests.
Sources
- Designing with Data (King et al., 2017)
- Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
Also known as
- Novelty and primacy effect check
- Learning-effect check
- Change-aversion check
- Novelty effect detection
- Long-term treatment effect estimation
- Post-period A/A analysis
- Time-staggered Treatments
- Learning effect measurement
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference