Description
Choose a smaller set of predictors by exhaustive, stepwise or filter search, judging subsets on held-out error or information criteria rather than training fit.
Subset selection reduces many candidate predictors to a smaller set for accuracy and interpretability. Best subset selection fits every model of each size and keeps the best by RSS or deviance, then chooses among sizes by cross-validation, Cp/AIC, BIC or adjusted R squared, never training RSS, which always falls as predictors are added. It is infeasible above about 40 predictors (over a million models at p = 20). Forward stepwise adds the most useful predictor each step (211 fits at p = 20) and works when n < p; backward needs n > p. Stepwise is greedy and may miss the best model. Machine-learning variants separate filters (distance, information, dependency, consistency measures, as in the Las Vegas filter) from wrappers (learner accuracy). Foster and Stine recommend a Bonferroni p-to-enter of 0.05/m. P-values from the selected model carry selection bias.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Vestigial
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Define the outcome and candidate predictors, discretising continuous inputs if using consistency filters.
- Choose a search: best subset (up to about 40 predictors), forward, backward or random.
- Choose an evaluation: a filter measure, or wrapper accuracy; for stepwise regression set p-to-enter at 0.05 divided by the number of candidates.
- For each model size, keep the best model by training fit.
- Estimate test error for each size by k-fold cross-validation, Cp/AIC, BIC or adjusted R squared.
- Pick the size with the lowest estimate, or the smallest within one standard error of it, and refit.
- Validate the chosen subset on held-out data and review it with domain experts; do not report naive p-values.
Sources
- Handbook of Data Mining (Ye, 2003)
- An Introduction to Statistical Learning with Applications in Python (James et al., 2023)
- Statistics for Business: Decision Making and Analysis (Stine & Foster, 2018)
- Applied Logistic Regression (Hosmer & Lemeshow, 2000)
- A User's Guide to Principal Components (Jackson, 1991)
- Multivariate Statistical Inference and Applications (Rencher, 1998)
- Regression and Key Driver Analysis (Sauro & Lewis, 2021)
- Advanced Quantitative Data Analysis (Cramer, 2003)
- The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)
- Statistical Design and Analysis of Experiments (Mason et al., 2003)
- Applied Predictive Modeling (Kuhn & Johnson, 2013)
- Cluster Analysis for Researchers (Romesburg, 1984)
- R for Marketing Research and Analytics (Chapman & Feit, 2019)
- Intermediate Statistics for Dummies (Rumsey, 2007)
- Practical Statistics for Data Scientists (Bruce et al., 2020)
Also known as
- Feature subset selection
- Feature selection
- Variable selection
- Best subset selection
- Stepwise regression
- Forward selection
- Backward elimination
- Las Vegas filter (LVF)
- Consistency-based selection
- One-standard-error rule
- Forward stepwise regression with a Bonferroni p-to-enter
Categories
- Analytics & Behaviour
- Statistics & Modelling