Prototype / Wizard-of-Oz A/B Test

Description

A randomised comparison of design variants using prototypes or human-simulated systems, measuring the same task and metrics as a live A/B test when shipping code is impossible.

The randomised A/B test is the gold standard for proving a design change caused improvement, but it needs shipped code and traffic. When shipping is impossible, Vivek Gopinath's playbook runs the comparison on low-fidelity prototypes or Wizard-of-Oz simulations with the same task and metrics. In the worked example, 600 participants are randomised to the old or new IA for 'Create a price alert for stock ABC', logging time to completion or abandonment, error type and clicks; completion rises from 62% to 81%, median completion time falls from 214s to 182s and median abandonment time rises from 46s to 71s. Decision bars and guardrails are set upfront and results feed a Cox survival model.

Attributes

Loss type
Active
Effort
Weeks

Steps

  1. Set decision bars and guardrails before the test (e.g. completion ≥ 80%, HR ≤ 0.80, no error increase elsewhere).
  2. Build control and variant prototypes (or a Wizard-of-Oz simulation) that differ only in the change under test.
  3. Randomise participants to arms with adequate sample size (the example uses N = 600).
  4. Run the identical task and log completion or abandonment time, error types and clicks.
  5. Compare completion, median completion time and median abandonment time, and fit a survival model.
  6. Issue a verdict against the bars and plan a confirmatory live test.

Sources

  • Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation

Also known as

  • Prototype A/B
  • Low-fidelity randomised comparison

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference