Leave-one-out cross-validation

Description

Estimates how well a model generalises by training on all but one case and testing on the excluded case, repeated for every case.

Evaluating a classifier or model on the data used to fit it overstates performance. LOO-CV fits the model n times, each time leaving one observation out and predicting it, then aggregates prediction errors. It uses nearly all data for training each time, which suits small UX datasets such as 50-300 labelled sessions or participants. Its estimates have low bias but can have high variance, and for large datasets 5- or 10-fold cross-validation is more efficient. Rigour requires that all preprocessing and feature selection occur inside each fold to avoid leakage, and that when participants contribute multiple observations, all of a participant's data are left out together (leave-one-subject-out).

Attributes

Loss type
Siloed
Stage
Post-launch
Effort
Hours

Steps

  1. Define the performance metric (accuracy, AUC, RMSE) and decide the unit to leave out (observation or participant).
  2. For each unit, fit the full pipeline (scaling, feature selection, model) on the remaining data.
  3. Predict the held-out unit and record the prediction.
  4. Aggregate predictions to compute performance metrics and a confusion matrix.
  5. Compare against a baseline (e.g. majority class or mean prediction) and against k-fold results for stability.

Sources

  • Pattern Classification (Richard Duda, Peter Hart & David Stork)
  • Practical Statistics for Data Scientists (Peter Bruce, Andrew Bruce & Peter Gedeck)
  • Methods Atlas (Vivek Gopinath)
  • Stone (1974) Cross-validatory choice and assessment of statistical predictions
  • Pattern Classification (Duda)
  • Practical Statistics for Data Scientists
  • Methods Atlas (Gopinath)
  • The Inquiry Series, Ch 54 — Linear Discriminant Analysis & Behavioral Classification
  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • LOO-CV
  • LOOCV
  • Jackknife validation

Categories

  • Analytics & Behaviour
  • Statistics & Modelling