Forecast Stage (Forward-Thinking Loop)

Description

Stage 4 of the forward-thinking loop: estimate rollout impact with Monte Carlo journey simulation, rank drivers with SHAP, and check emotional sustainability with diary trajectories.

Forecasting bridges small-sample evidence and large-scale rollout by producing a distribution of outcomes rather than a single number. Monte Carlo samples step-level success probabilities across 10,000 virtual users to yield a completion distribution with a 95% interval, and supports what-if scenarios. A gradient boosting model trained on a 70/30 split with task time, error counts, SNR, retries and TLX features is decomposed with SHAP to rank drivers. Qualitative forecasting via diary studies arrays sentiment trajectories (clarity, trust, frustration per session) into patterns such as 'fast trust gain', 'fragile success' and 'churn risk', since negative sentiment drift predicts future drop-off even when completion rises. Rigour comes from stated assumptions (stationarity, approximate independence, like-for-like traffic, no leakage), sensitivity checks (best/mid/worst ±10–20%, segment splits, volume scaling) and calibration plots.

Attributes

Loss type
Community-proposed
Data
Mixed
Effort
Weeks

Steps

  1. Collect empirical step probabilities from the variant (e.g. Find 81%, Set Rule 75%, OTP 85%, Confirm 95%).
  2. Simulate 10,000 virtual journeys, sampling success or failure at each step, and report mean completion with a 95% CI (example: 49% [46%, 52%]).
  3. Run what-if scenarios on the weakest step (OTP improved to 92% gives 54% [51%, 57%]) and sensitivity checks of ±10–20%, segment splits and volume scaling.
  4. Train a gradient boosting model on a 70/30 split using pre-outcome task features, validate on holdout data and check calibration with reliability plots.
  5. Compute SHAP values to rank drivers (e.g. mistake rate 0.33, SNR 0.27, OTP time 0.18, frustration 0.13, retries 0.09).
  6. Array diary entries into trajectories (session index, clarity −2..+2, trust 1..5, frustration 0..100, success, quote) and cluster them into patterns.
  7. Link top SHAP drivers to trajectory clusters and present Monte Carlo curves alongside two or three curated trajectories.

Sources

  • Vivek Gopinath (2025), Forward-Thinking UX Research Playbook: A Mixed-Methods, Decision-Ready Approach from Prediction to Validation

Also known as

  • Stage 4: Forecast with Simulation and Explainability

Categories

  • Community-Proposed Frameworks