Counterfactual Probing

Description

Interview probes asking participants to contrast before and after a change, or an alternative design, to explain the mechanism behind a measured effect.

Quantitative evidence can show that risk fell but not why. Counterfactual probing asks participants to reason about the alternative world: 'If we labelled it Price Alerts at top level, would you choose differently? Why?', or, after a change, how the old and new experiences differ. In the playbook, quotes such as 'Before, I always checked Notifications first' and 'Now Alerts is right where I expect it under Markets' explain the mechanism behind a hazard ratio of 0.68: fewer mistakes and better mental-model alignment. Responses are written into mechanism memos linking minimised QCA paths, quotes and HRs. Rigour comes from pairing probes with measured effects rather than relying on stated preference alone, since participants' counterfactual predictions are fallible.

Attributes

Loss type
Latent
Stage
Evaluation
Effort
Days

Steps

  1. Identify the measured effect or design change whose mechanism needs explaining.
  2. After the task, ask participants what they expected and whether an alternative label or structure would change their choice.
  3. For post-change sessions, ask participants to contrast the old and new experience in their own words.
  4. Code answers for mechanism (e.g. mental-model alignment, reduced time pressure, increased confidence).
  5. Write mechanism memos linking codes to the quantitative effect.
  6. Treat stated counterfactuals as supporting evidence, not proof, and confirm with behavioural data.

Sources

  • Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation
  • Vivek Gopinath (2025), Tree Testing and Error Taxonomy: A Complete Chapter for UX Researchers
  • Causal Inference
  • Counterfactuals (David Lewis)
  • The Book of Why (Pearl & Mackenzie)

Also known as

  • Counterfactual interview probes
  • Qualitative proof
  • Potential outcomes thinking
  • Counterfactual analysis
  • Counterfactual probing
  • Counterfactual Reasoning

Categories

  • User Research
  • Interviewing & Elicitation