Description
Compare a behaviour rate in stable periods before and after a release with a two-proportion test, then reason through time, experience, data-quality and population confounds against a care threshold.
When an A/B test is impossible, teams still need a defensible read of a release. The target behaviour is reframed as a binary rate (conversion, exit, reached a page, event occurred). A long, stable before period without outages, holidays or other changes is compared with about two weeks or 5,000 visitors after, using a two-proportion (N-1 chi-square) test with a confidence interval and a chart of the rate over time with the release marked. A smallest-change-worth-caring-about threshold is set first. Then four non-product explanations are checked: time effects (season, weekday), growing user experience, data availability or quality changes (sharp and often hidden), and population composition, countered by following a fixed group. If confounds could push the effect below the threshold, stop; if not, treat it as an initial estimate and confirm with a model using those factors as controls.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Name the behaviour the change should affect and reframe it as a rate (for example unique visitors reaching the target page divided by all unique visitors).
- Set the smallest change you would care about before looking at the data.
- Choose a before period as long and stable as possible, free of outages, holidays, campaigns and other releases.
- Collect after data (start at about two weeks or 5,000 visitors), segmenting to target users if the change affects only some.
- Compare rates with a two-proportion test, report the confidence interval and chart the rate over time with the release marked.
- Check confounds: time effects, user experience effects, data availability or quality changes, and population composition (follow a fixed group).
- Stop if confounds could push the effect below the threshold; otherwise confirm with a statistical model that uses those factors as controls.
Sources
- Designing for Behavior Change (Wendel, 2014)
- Practical Web Analytics for User Experience (Beasley, 2013)
- Handbook of Usability Testing (Rubin & Chisnell, 2008)
Also known as
- Informal pre-post impact analysis
- Pre-post gut check
- Before-after confound checklist
- Before-and-after release comparison with analytics rates
- Measuring the effects of changes
- Reframe as a rate
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference