Applicability-Domain Check for Predictive Models

Description

Train a classifier to tell real training rows from a permuted copy of them, then use its membership probability to flag new cases whose predictions are likely extrapolations.

Predictive models assume new cases come from the same data-generating process as the training set. Extrapolation includes values outside training ranges and also empty interior regions of a high-dimensional predictor space. Knowing how both data sets were produced is the first defence. A quick global check projects training and new data onto principal components and looks for overlap, but two-dimensional projections can hide sparse pockets. For a case-level check Kuhn and Johnson adapt Hastie and colleagues: rather than sampling uniformly across predictor ranges, they permute each predictor independently, which spreads points across the space while keeping only allowable values, and they restrict this to the top 20 or so important predictors. A classifier trained to separate real from permuted rows then scores each new case by its probability of resembling the training data. Low scores mark predictions to treat with caution.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Fit the predictive model, compute variable importance and keep the top predictors (for example 20).
  2. Create a random version of the training set by permuting each of those predictors independently.
  3. Stack the real and permuted rows and label them training versus random.
  4. Train a classifier (for example bagged trees) to separate the two.
  5. Predict the probability of training-set membership for each new case, and overlay training and new data on principal components as a global check.
  6. Flag low-probability cases as extrapolations, and route them to fallback behaviour or human review.

Sources

  • Applied Predictive Modeling (Kuhn & Johnson, 2013)

Also known as

  • Applicability domain
  • Extrapolation detection
  • Training-set membership model
  • Permuted-predictor similarity score

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning