Description
Score each candidate predictor's relationship with an outcome using a statistic matched to the data types, plot effect against significance, and check scores against a permutation baseline before shortlisting.
Before or instead of fitting a model, each predictor can be scored on its own as a triage step for large predictor sets. The statistic depends on data types: correlation, a LOESS pseudo R squared for curved relationships, or the maximal information coefficient for arbitrary shapes when both are numeric; a t-test on the difference in means for a binary predictor, displayed as a volcano plot of -log10(p) against effect so large-but-noisy and small-but-precise effects are distinguished; the area under the ROC curve treating the predictor itself as the score for a categorical outcome; and odds ratios with Fisher's exact test for two binary variables. ReliefF works for any type and can reflect interactions by comparing nearest same-class and different-class neighbours. Because its scores lack a natural threshold, the outcome labels are shuffled many times to build a null distribution, and observed scores are expressed in standard deviations above chance.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Unmigrated
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Match the score to the outcome and predictor types: correlation, LOESS pseudo R squared or MIC; t-test with volcano plot; per-predictor ROC AUC; odds ratio with Fisher's exact test; gain ratio; or ReliefF.
- Compute the score for every candidate predictor on the training data only.
- Plot effect size against significance (volcano plot) or compare several scores in a scatter-plot matrix.
- For scores without a natural threshold, permute the outcome labels many times (for example 500) and recompute to build a null distribution.
- Shortlist predictors that stand well above the null, applying multiplicity control, then examine them with plots and domain knowledge.
- Remove redundant, highly correlated survivors, look for interactions the univariate scores miss, and repeat any filtering inside resampling if it feeds a model.
Sources
- Applied Predictive Modeling (Kuhn & Johnson, 2013)
Also known as
- Univariate filter scores
- Volcano plot of predictor effects
- Per-predictor ROC AUC
- ReliefF with permutation test
- Maximal information coefficient (MIC)
- LOESS pseudo R squared
- Filter-based feature selection
Categories
- Analytics & Behaviour
- Computational & Machine Learning