Description
Tests whether models predicting UX outcomes such as churn, conversion or satisfaction generalise to new data, using cross-validation, holdout sets and calibration checks.
Predictive model validation asks whether a model's performance will hold for data it has not seen. Following Kuhn and Johnson, the analyst splits data into training and test sets, or uses k-fold or time-based cross-validation, tunes models only within training folds, and evaluates on held-out data with metrics appropriate to the task: AUC, precision and recall for imbalanced outcomes, RMSE for continuous ones, and calibration for probability estimates. Performance is compared with simple baselines such as the base rate or a logistic model. Leakage checks ensure that predictors do not contain information from after the outcome. Rigour comes from pre-specifying the validation scheme, reporting uncertainty in performance estimates, and monitoring for drift after deployment.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Validation protocol for UX predictive models:
- Split data: 70% training, 30% held-out test.
- (2) Within training: cross-validation (10-fold CV) for model selection and hyperparameter tuning.
- (3) Evaluate on held-out test set only once.
- (4) Report: both training CV accuracy AND held-out test accuracy. Large gap (> 5pp) = overfitting.
- (5) Temporal validation: for time-series data, use time-based split (train on months 1–9, test on months 10–12).
- (6) Subgroup analysis: validate performance separately for key user segments.
Sources
- Applied Predictive Modeling — Kuhn & Johnson
- Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
Also known as
- Out-of-sample validation
- Cross-validation for UX models
- Model performance assessment
Categories
- Analytics & Behaviour
- Statistics & Modelling