Description
Techniques for modelling rare but important events, such as errors, cancellations or accessibility failures, without the model ignoring them.
When only 2% of sessions end in a critical error, a model that predicts "no error" is 98% accurate and useless. Imbalanced data strategies rebalance the training data by undersampling the common class, oversampling or up-weighting the rare class, or generating synthetic rare cases (SMOTE). Cost-based classification sets thresholds by the real cost of each mistake. Bruce and Bruce stress evaluating with precision, recall and lift rather than accuracy.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Unmigrated
- Goal
- Predict performance
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Quantify the class imbalance and the cost of missing a rare case.
- Try undersampling, oversampling, weighting or SMOTE on the training set only.
- Fit the model and evaluate on an untouched test set with recall, precision and AUC.
- Set the decision threshold using the cost of errors.
- Review predicted rare cases qualitatively.
Sources
- Practical Statistics for Data Scientists (Bruce & Bruce)
Also known as
- Rare class modelling
- Class imbalance handling
Categories
- Analytics & Behaviour
- Computational & Machine Learning