Feature subset and stepwise selection

Description

Choose a smaller set of predictors by exhaustive, stepwise or filter search, judging subsets on held-out error or information criteria rather than training fit.

Subset selection reduces many candidate predictors to a smaller set for accuracy and interpretability. Best subset selection fits every model of each size and keeps the best by RSS or deviance, then chooses among sizes by cross-validation, Cp/AIC, BIC or adjusted R squared, never training RSS, which always falls as predictors are added. It is infeasible above about 40 predictors (over a million models at p = 20). Forward stepwise adds the most useful predictor each step (211 fits at p = 20) and works when n < p; backward needs n > p. Stepwise is greedy and may miss the best model. Machine-learning variants separate filters (distance, information, dependency, consistency measures, as in the Las Vegas filter) from wrappers (learner accuracy). Foster and Stine recommend a Bonferroni p-to-enter of 0.05/m. P-values from the selected model carry selection bias.

Attributes

Loss type
Vestigial
Effort
Days

Steps

  1. Define the outcome and candidate predictors, discretising continuous inputs if using consistency filters.
  2. Choose a search: best subset (up to about 40 predictors), forward, backward or random.
  3. Choose an evaluation: a filter measure, or wrapper accuracy; for stepwise regression set p-to-enter at 0.05 divided by the number of candidates.
  4. For each model size, keep the best model by training fit.
  5. Estimate test error for each size by k-fold cross-validation, Cp/AIC, BIC or adjusted R squared.
  6. Pick the size with the lowest estimate, or the smallest within one standard error of it, and refit.
  7. Validate the chosen subset on held-out data and review it with domain experts; do not report naive p-values.

Sources

  • Handbook of Data Mining (Ye, 2003)
  • An Introduction to Statistical Learning with Applications in Python (James et al., 2023)
  • Statistics for Business: Decision Making and Analysis (Stine & Foster, 2018)
  • Applied Logistic Regression (Hosmer & Lemeshow, 2000)
  • A User's Guide to Principal Components (Jackson, 1991)
  • Multivariate Statistical Inference and Applications (Rencher, 1998)
  • Regression and Key Driver Analysis (Sauro & Lewis, 2021)
  • Advanced Quantitative Data Analysis (Cramer, 2003)
  • The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)
  • Statistical Design and Analysis of Experiments (Mason et al., 2003)
  • Applied Predictive Modeling (Kuhn & Johnson, 2013)
  • Cluster Analysis for Researchers (Romesburg, 1984)
  • R for Marketing Research and Analytics (Chapman & Feit, 2019)
  • Intermediate Statistics for Dummies (Rumsey, 2007)
  • Practical Statistics for Data Scientists (Bruce et al., 2020)

Also known as

  • Feature subset selection
  • Feature selection
  • Variable selection
  • Best subset selection
  • Stepwise regression
  • Forward selection
  • Backward elimination
  • Las Vegas filter (LVF)
  • Consistency-based selection
  • One-standard-error rule
  • Forward stepwise regression with a Bonferroni p-to-enter

Categories

  • Analytics & Behaviour
  • Statistics & Modelling