Predictive Model Validation

Description

Tests whether models predicting UX outcomes such as churn, conversion or satisfaction generalise to new data, using cross-validation, holdout sets and calibration checks.

Predictive model validation asks whether a model's performance will hold for data it has not seen. Following Kuhn and Johnson, the analyst splits data into training and test sets, or uses k-fold or time-based cross-validation, tunes models only within training folds, and evaluates on held-out data with metrics appropriate to the task: AUC, precision and recall for imbalanced outcomes, RMSE for continuous ones, and calibration for probability estimates. Performance is compared with simple baselines such as the base rate or a logistic model. Leakage checks ensure that predictors do not contain information from after the outcome. Rigour comes from pre-specifying the validation scheme, reporting uncertainty in performance estimates, and monitoring for drift after deployment.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Validation protocol for UX predictive models:
  2. Split data: 70% training, 30% held-out test.
  3. (2) Within training: cross-validation (10-fold CV) for model selection and hyperparameter tuning.
  4. (3) Evaluate on held-out test set only once.
  5. (4) Report: both training CV accuracy AND held-out test accuracy. Large gap (> 5pp) = overfitting.
  6. (5) Temporal validation: for time-series data, use time-based split (train on months 1–9, test on months 10–12).
  7. (6) Subgroup analysis: validate performance separately for key user segments.

Sources

  • Applied Predictive Modeling — Kuhn & Johnson
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)

Also known as

  • Out-of-sample validation
  • Cross-validation for UX models
  • Model performance assessment

Categories

  • Analytics & Behaviour
  • Statistics & Modelling