Description
A randomised comparison of design variants using prototypes or human-simulated systems, measuring the same task and metrics as a live A/B test when shipping code is impossible.
The randomised A/B test is the gold standard for proving a design change caused improvement, but it needs shipped code and traffic. When shipping is impossible, Vivek Gopinath's playbook runs the comparison on low-fidelity prototypes or Wizard-of-Oz simulations with the same task and metrics. In the worked example, 600 participants are randomised to the old or new IA for 'Create a price alert for stock ABC', logging time to completion or abandonment, error type and clicks; completion rises from 62% to 81%, median completion time falls from 214s to 182s and median abandonment time rises from 46s to 71s. Decision bars and guardrails are set upfront and results feed a Cox survival model.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Weeks
Steps
- Set decision bars and guardrails before the test (e.g. completion ≥ 80%, HR ≤ 0.80, no error increase elsewhere).
- Build control and variant prototypes (or a Wizard-of-Oz simulation) that differ only in the change under test.
- Randomise participants to arms with adequate sample size (the example uses N = 600).
- Run the identical task and log completion or abandonment time, error types and clicks.
- Compare completion, median completion time and median abandonment time, and fit a survival model.
- Issue a verdict against the bars and plan a confirmatory live test.
Sources
- Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation
Also known as
- Prototype A/B
- Low-fidelity randomised comparison
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference