Description
Compare two live design variants with real traffic to see which performs better on your chosen metrics.
Teams run design variations against real users so that shipping decisions rest on evidence, not instinct. They split live traffic between the original design, called the control, and one or more variants that usually change a single element such as a button, headline or piece of copy, then compare the versions against pre-agreed business metrics like conversion, click-through and retention. A sound test starts from a hypothesis grounded in user research, tracks guardrail metrics alongside the primary one, and runs until the required sample size is reached, typically at least a week or two. The method suits busy, high-traffic products where small usability gains compound, but it reveals what changed rather than why, so teams combine it with qualitative research to interpret the results.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Write a hypothesis and choose one primary metric, plus guardrail metrics.
- Set the minimum detectable effect and compute the required sample size with a power analysis (typically 80 percent power, alpha 0.05).
- Implement random assignment at the appropriate unit (user, account or session) and validate with an A/A test if new to the platform.
- Run for the precomputed duration, at least one full weekly cycle, without stopping early unless using sequential methods.
- Check sample ratio mismatch and data quality before analysis.
- Analyse the primary metric with confidence intervals and review guardrails and pre-planned segments.
- Decide, document and share results, including null results.
Sources
- Designing with Data
- Quantifying the User Experience
- Trustworthy Online Controlled Experiments (Kohavi, Tang & Xu)
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Quantifying the User Experience (Sauro & Lewis)
- Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
Videos
Related talks
Also known as
- A/B test
- AB testing
- split testing
- split test
- bucket testing
- Online controlled experiment
- Randomised controlled experiment
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference