Pre-post release comparison with confound checklist

Description

Compare a behaviour rate in stable periods before and after a release with a two-proportion test, then reason through time, experience, data-quality and population confounds against a care threshold.

When an A/B test is impossible, teams still need a defensible read of a release. The target behaviour is reframed as a binary rate (conversion, exit, reached a page, event occurred). A long, stable before period without outages, holidays or other changes is compared with about two weeks or 5,000 visitors after, using a two-proportion (N-1 chi-square) test with a confidence interval and a chart of the rate over time with the release marked. A smallest-change-worth-caring-about threshold is set first. Then four non-product explanations are checked: time effects (season, weekday), growing user experience, data availability or quality changes (sharp and often hidden), and population composition, countered by following a fixed group. If confounds could push the effect below the threshold, stop; if not, treat it as an initial estimate and confirm with a model using those factors as controls.

Attributes

Loss type
Active
Stage
Post-launch
Effort
Hours

Steps

  1. Name the behaviour the change should affect and reframe it as a rate (for example unique visitors reaching the target page divided by all unique visitors).
  2. Set the smallest change you would care about before looking at the data.
  3. Choose a before period as long and stable as possible, free of outages, holidays, campaigns and other releases.
  4. Collect after data (start at about two weeks or 5,000 visitors), segmenting to target users if the change affects only some.
  5. Compare rates with a two-proportion test, report the confidence interval and chart the rate over time with the release marked.
  6. Check confounds: time effects, user experience effects, data availability or quality changes, and population composition (follow a fixed group).
  7. Stop if confounds could push the effect below the threshold; otherwise confirm with a statistical model that uses those factors as controls.

Sources

  • Designing for Behavior Change (Wendel, 2014)
  • Practical Web Analytics for User Experience (Beasley, 2013)
  • Handbook of Usability Testing (Rubin & Chisnell, 2008)

Also known as

  • Informal pre-post impact analysis
  • Pre-post gut check
  • Before-after confound checklist
  • Before-and-after release comparison with analytics rates
  • Measuring the effects of changes
  • Reframe as a rate

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference