Novelty, primacy and long-term effect estimation

Description

Check whether an experiment's effect decays (novelty) or grows (learning, change aversion) over time, and use cohort, post-period or staggered designs to estimate the long-term effect.

Short A/B tests assume a constant treatment effect, but visible changes often provoke novelty clicks or change aversion that fade as users adapt. Per-day effect plots, using each day's own visitors rather than cumulative plots that create false trends, show decline (novelty) or growth (primacy, learning). Segmenting by usage frequency helps: in Google's 2010 search redesign, high-frequency users who met the change most looked better early, and others converged after a longer run; large visual changes may need about six weeks. New users are a check: if a change wins only for them, suspect novelty. For durable estimates, cohort analysis tracks stable signed-in users; post-period analysis switches the treatment off and measures the residual learned effect (Google found users learned to click fewer ads as ad load rose); time-staggered treatments start a second copy later and declare convergence when the two agree.

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. Plot the treatment effect for each day using that day's visitors only, and plot the effect for the cohort first exposed on day one or two.
  2. Segment by usage frequency and tenure: if high-frequency users already look better, exposure and learning are driving the early result.
  3. Compare new users: a change that is flat or negative for them is not explained by habituation; one that wins only for them suggests novelty.
  4. If a trend appears, extend the run until the effect stabilises (large visual changes may need about six weeks) and make longer durations the default for visual changes.
  5. When the long-term effect matters, choose a design: stable signed-in cohort reweighted by pre-experiment strata, post-period analysis after switching the treatment off, or a time-staggered second treatment copy.
  6. Test convergence, separate user-learned from system-learned effects, and model learned effects to forecast long-term impact from future short tests.

Sources

  • Designing with Data (King et al., 2017)
  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)

Also known as

  • Novelty and primacy effect check
  • Learning-effect check
  • Change-aversion check
  • Novelty effect detection
  • Long-term treatment effect estimation
  • Post-period A/A analysis
  • Time-staggered Treatments
  • Learning effect measurement

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference