Imbalanced Data Strategies (Undersampling, Oversampling, SMOTE)

Description

Techniques for modelling rare but important events, such as errors, cancellations or accessibility failures, without the model ignoring them.

When only 2% of sessions end in a critical error, a model that predicts "no error" is 98% accurate and useless. Imbalanced data strategies rebalance the training data by undersampling the common class, oversampling or up-weighting the rare class, or generating synthetic rare cases (SMOTE). Cost-based classification sets thresholds by the real cost of each mistake. Bruce and Bruce stress evaluating with precision, recall and lift rather than accuracy.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Hours

Steps

  1. Quantify the class imbalance and the cost of missing a rare case.
  2. Try undersampling, oversampling, weighting or SMOTE on the training set only.
  3. Fit the model and evaluate on an untouched test set with recall, precision and AUC.
  4. Set the decision threshold using the cost of errors.
  5. Review predicted rare cases qualitatively.

Sources

  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • Rare class modelling
  • Class imbalance handling

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning