Model-Free Predictor Importance Screening

Description

Score each candidate predictor's relationship with an outcome using a statistic matched to the data types, plot effect against significance, and check scores against a permutation baseline before shortlisting.

Before or instead of fitting a model, each predictor can be scored on its own as a triage step for large predictor sets. The statistic depends on data types: correlation, a LOESS pseudo R squared for curved relationships, or the maximal information coefficient for arbitrary shapes when both are numeric; a t-test on the difference in means for a binary predictor, displayed as a volcano plot of -log10(p) against effect so large-but-noisy and small-but-precise effects are distinguished; the area under the ROC curve treating the predictor itself as the score for a categorical outcome; and odds ratios with Fisher's exact test for two binary variables. ReliefF works for any type and can reflect interactions by comparing nearest same-class and different-class neighbours. Because its scores lack a natural threshold, the outcome labels are shuffled many times to build a null distribution, and observed scores are expressed in standard deviations above chance.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Match the score to the outcome and predictor types: correlation, LOESS pseudo R squared or MIC; t-test with volcano plot; per-predictor ROC AUC; odds ratio with Fisher's exact test; gain ratio; or ReliefF.
  2. Compute the score for every candidate predictor on the training data only.
  3. Plot effect size against significance (volcano plot) or compare several scores in a scatter-plot matrix.
  4. For scores without a natural threshold, permute the outcome labels many times (for example 500) and recompute to build a null distribution.
  5. Shortlist predictors that stand well above the null, applying multiplicity control, then examine them with plots and domain knowledge.
  6. Remove redundant, highly correlated survivors, look for interactions the univariate scores miss, and repeat any filtering inside resampling if it feeds a model.

Sources

  • Applied Predictive Modeling (Kuhn & Johnson, 2013)

Also known as

  • Univariate filter scores
  • Volcano plot of predictor effects
  • Per-predictor ROC AUC
  • ReliefF with permutation test
  • Maximal information coefficient (MIC)
  • LOESS pseudo R squared
  • Filter-based feature selection

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning