Prove Stage (Forward-Thinking Loop)

Description

Stage 3 of the forward-thinking loop: show a design change caused improvement using A/B tests, Cox survival models, QCA and counterfactual probing, with fallbacks when randomisation is impossible.

Proving answers 'Did our change cause improvement, or was it coincidence?' by showing that changing X changes Y and ruling out chance, seasonality and user mix. The playbook gives a selection rule: randomised A/B as gold standard; prototype or Wizard-of-Oz A/B when shipping code is impossible; Cox proportional hazards when the question is when users abandon, not just whether; Rubin's potential outcomes with matched cohorts when randomisation is impossible; Bradford Hill criteria as a principled checklist for observational data. Small-N moderated data are analysed with crisp-set QCA for sufficient condition combinations, and counterfactual probing in interviews explains the mechanism behind the hazard ratio. Decision bars are set upfront (completion ≥ 80%, HR ≤ 0.80, no error increase elsewhere). Rigour includes testing the proportional hazards assumption with Schoenfeld residuals and always checking guardrails.

Attributes

Loss type
Community-proposed
Data
Mixed
Effort
Weeks

Steps

  1. Set decision bars before the test, e.g. completion ≥ 80% (up from 62%), hazard ratio of abandonment ≤ 0.80, and no increase in error rates elsewhere.
  2. Randomise participants to control and variant (the worked example uses N = 600) on the same task, logging time to completion or abandonment, error type and clicks.
  3. Compare completion, median completion time and median abandonment time between arms (62% vs 81%; 214s vs 182s; 46s vs 71s).
  4. Fit a Cox model h(t|X) = h0(t)·exp(β1·Variant + β2·SNR + β3·Mistake rate), report HRs with 95% CIs, and test proportional hazards with Schoenfeld residuals.
  5. For small moderated samples, run crisp-set QCA on calibrated conditions (e.g. Low SNR, High Mistakes, High TLX, Timed OTP) to find sufficient paths to abandonment.
  6. Use counterfactual probing in interviews to explain the mechanism, and write mechanism memos linking the QCA path, quotes and HRs.
  7. If randomisation is impossible, match cohorts under the Rubin model to estimate ATE and use Bradford Hill criteria (temporality, strength, consistency, plausibility) as a narrative check.
  8. Deliver the A/B table, Cox table, mechanism quotes, guardrail checks and a decision bar verdict.

Sources

  • Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation

Also known as

  • Stage 3: Prove with Experiments and Causal Models

Categories

  • Community-Proposed Frameworks