Description
Estimates how well a model generalises by training on all but one case and testing on the excluded case, repeated for every case.
Evaluating a classifier or model on the data used to fit it overstates performance. LOO-CV fits the model n times, each time leaving one observation out and predicting it, then aggregates prediction errors. It uses nearly all data for training each time, which suits small UX datasets such as 50-300 labelled sessions or participants. Its estimates have low bias but can have high variance, and for large datasets 5- or 10-fold cross-validation is more efficient. Rigour requires that all preprocessing and feature selection occur inside each fold to avoid leakage, and that when participants contribute multiple observations, all of a participant's data are left out together (leave-one-subject-out).
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Define the performance metric (accuracy, AUC, RMSE) and decide the unit to leave out (observation or participant).
- For each unit, fit the full pipeline (scaling, feature selection, model) on the remaining data.
- Predict the held-out unit and record the prediction.
- Aggregate predictions to compute performance metrics and a confusion matrix.
- Compare against a baseline (e.g. majority class or mean prediction) and against k-fold results for stability.
Sources
- Pattern Classification (Richard Duda, Peter Hart & David Stork)
- Practical Statistics for Data Scientists (Peter Bruce, Andrew Bruce & Peter Gedeck)
- Methods Atlas (Vivek Gopinath)
- Stone (1974) Cross-validatory choice and assessment of statistical predictions
- Pattern Classification (Duda)
- Practical Statistics for Data Scientists
- Methods Atlas (Gopinath)
- The Inquiry Series, Ch 54 — Linear Discriminant Analysis & Behavioral Classification
- Practical Statistics for Data Scientists (Bruce & Bruce)
Also known as
- LOO-CV
- LOOCV
- Jackknife validation
Categories
- Analytics & Behaviour
- Statistics & Modelling