Description
Fit several model types on identical resamples, find the best achievable performance, then use paired comparisons of resampled metrics to keep the simplest model that is not meaningfully worse.
No model type is best for every problem, so Kuhn and Johnson recommend a three-step strategy. First fit flexible, opaque models such as boosted trees or support vector machines to find the performance ceiling. Then fit simpler, more interpretable models on the same resamples. Finally adopt the simplest model that reasonably approaches the ceiling, weighing interpretability, prediction speed and deployment. Because every model is evaluated on identical resampled data sets, the metrics are paired, so a paired t-test or confidence intervals on per-resample differences, adjusted for multiplicity when many pairs are compared, tell whether a difference is real. Differences must also be judged for practical meaning: a credit model comparison found a 0.1 per cent accuracy difference with an interval spanning zero, so the logistic regression was kept.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Create one set of resamples (for example five repeats of 10-fold cross-validation) and reuse it for every candidate model.
- Tune and evaluate flexible, low-interpretability models to establish the performance ceiling.
- Tune and evaluate simpler, more interpretable models on the same resamples.
- Compute per-resample differences in the chosen metric between pairs of models.
- Form confidence intervals or paired tests for the mean differences, adjusting for multiplicity.
- Choose the simplest model whose performance is not meaningfully below the ceiling, considering interpretability and deployment, and confirm it on the held-out test set.
Sources
- Applied Predictive Modeling (Kuhn & Johnson, 2013)
Also known as
- Paired resampling comparison of predictive models (performance-ceiling strategy)
- Comparing models on common resamples
- Resample-paired t-test of models
- Performance ceiling then simplest adequate model
Categories
- Analytics & Behaviour
- Computational & Machine Learning