Description
Designs online experiments around a pre-specified hierarchy of primary, secondary and guardrail metrics, with power analysis, so decisions reflect user experience and long-term value, not conversion alone.
Many A/B tests optimise a single short-term conversion metric, which can reward dark patterns and miss harm to retention, trust or support load. Rigorous experiment design pre-registers a metric hierarchy: a primary decision metric (or overall evaluation criterion) that predicts long-term value, secondary metrics that explain the mechanism, and guardrail metrics that must not degrade, such as latency, error rates, unsubscribes, complaints or retention. A power analysis sets sample size and duration for the primary metric's minimum detectable effect, and the analysis plan specifies decision rules before data arrive. King, Churchill and Tan add the UX perspective: grounding hypotheses in qualitative insight and treating experiments as part of a learning agenda. Holdout groups and longer-term follow-ups detect novelty effects and delayed harm.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Before any A/B test:
- Primary metric — what single metric determines ship/no-ship?
- (2) Secondary metrics — what additional signals support the decision?
- (3) Guardrails — what metrics must not worsen (latency, error rate, NPS)?
- (4) Power calculation for primary metric. Run the test. Report: did primary metric move? Did guardrails hold? Did secondary metrics tell a consistent story?
Sources
- Designing with Data — O'Reilly
- Trustworthy Online Controlled Experiments (Ron Kohavi, Diane Tang & Ya Xu)
Also known as
- Metric-hierarchy A/B testing
- Guardrail metric design
- Overall evaluation criterion design
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference