Description
Stage 3 of the forward-thinking loop: show a design change caused improvement using A/B tests, Cox survival models, QCA and counterfactual probing, with fallbacks when randomisation is impossible.
Proving answers 'Did our change cause improvement, or was it coincidence?' by showing that changing X changes Y and ruling out chance, seasonality and user mix. The playbook gives a selection rule: randomised A/B as gold standard; prototype or Wizard-of-Oz A/B when shipping code is impossible; Cox proportional hazards when the question is when users abandon, not just whether; Rubin's potential outcomes with matched cohorts when randomisation is impossible; Bradford Hill criteria as a principled checklist for observational data. Small-N moderated data are analysed with crisp-set QCA for sufficient condition combinations, and counterfactual probing in interviews explains the mechanism behind the hazard ratio. Decision bars are set upfront (completion ≥ 80%, HR ≤ 0.80, no error increase elsewhere). Rigour includes testing the proportional hazards assumption with Schoenfeld residuals and always checking guardrails.
Attributes
- Discipline
- Community-Proposed Frameworks
- Loss type
- Community-proposed
- Tested
- Not yet tested
- Goal
- Prove causation
- Stage
- Evaluation, Post-launch
- Data
- Mixed
- Effort
- Weeks
Steps
- Set decision bars before the test, e.g. completion ≥ 80% (up from 62%), hazard ratio of abandonment ≤ 0.80, and no increase in error rates elsewhere.
- Randomise participants to control and variant (the worked example uses N = 600) on the same task, logging time to completion or abandonment, error type and clicks.
- Compare completion, median completion time and median abandonment time between arms (62% vs 81%; 214s vs 182s; 46s vs 71s).
- Fit a Cox model h(t|X) = h0(t)·exp(β1·Variant + β2·SNR + β3·Mistake rate), report HRs with 95% CIs, and test proportional hazards with Schoenfeld residuals.
- For small moderated samples, run crisp-set QCA on calibrated conditions (e.g. Low SNR, High Mistakes, High TLX, Timed OTP) to find sufficient paths to abandonment.
- Use counterfactual probing in interviews to explain the mechanism, and write mechanism memos linking the QCA path, quotes and HRs.
- If randomisation is impossible, match cohorts under the Rubin model to estimate ATE and use Bradford Hill criteria (temporality, strength, consistency, plausibility) as a narrative check.
- Deliver the A/B table, Cox table, mechanism quotes, guardrail checks and a decision bar verdict.
Sources
- Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation
Also known as
- Stage 3: Prove with Experiments and Causal Models
Categories
- Community-Proposed Frameworks