Resampling the Whole Feature-Selection Process (Nested Resampling)

Description

Place every data-driven step, including predictor selection and tuning, inside an outer resampling loop so performance estimates include the optimism of the selection itself.

If predictors are chosen using all the training data and the chosen model is then cross-validated, the estimate ignores selection and can look perfect on pure noise. Guyon and colleagues reported 100% leave-one-out accuracy on 62 patients with 2,000 gene predictors; Ambroise and McLachlan scrambled the labels and obtained the same zero error, proving the estimate an artefact. The fix is an outer loop: in each resample, rank predictors, fit models for each subset size (tuning in an inner loop), and predict the held-back samples; aggregate across resamples to choose the subset size; only then rank and fit on the full training set. Castaldi and colleagues found 64% of published classification analyses skipped this. Scrambling labels and expecting no-information performance is a simple sanity check.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Set up an outer resampling scheme, for example five repeats of 10-fold cross-validation.
  2. Within each outer training portion, fit on all predictors, rank them, and for each candidate subset size refit (tuning with an inner resampling loop) and predict the outer held-back samples.
  3. Aggregate held-back performance across resamples into a profile over subset sizes and record how often each predictor is selected.
  4. Choose the subset size from that profile.
  5. Rank predictors and fit the final model on the whole training set.
  6. Confirm on an independent test set where possible, and rerun with scrambled labels, expecting performance at the no-information rate.

Sources

  • Applied Predictive Modeling (Kuhn & Johnson, 2013)

Also known as

  • Outer-loop resampling of feature selection
  • Nested resampling for feature selection
  • External cross-validation
  • Scrambled-label check
  • Selection-bias check

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning