Cost-sensitive decision thresholds and expected-loss rules

Description

Set a classifier's decision threshold from the costs of misses and false alarms, not 0.5, and choose models by expected cost, deferring uncertain cases to humans.

Inference (estimating class probabilities) is separated from decision. Stakeholders specify a loss matrix of costs for each true versus assigned class. For two classes, if a miss costs c1 and a false alarm c0, expected cost is minimised by acting whenever the predicted probability exceeds c0/(c0+c1), not 0.5. Models are compared by expected cost on held-out data, including trivial rules (flag none, flag all), alongside calibration checks such as the Brier score. A reject option defers cases whose largest posterior falls below a threshold to another process, such as human review. When a model is trained on rebalanced data, posteriors are rescaled by population over training priors. Keeping posteriors allows the loss matrix to change without retraining. Ridgeway's school-dropout example judged a miss 20 times worse than a false flag, giving a threshold of 0.047.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Agree with stakeholders the cost of each error type (and of deferral), and record the reasoning.
  2. Fit probability models and check calibration with a calibration plot and Brier score.
  3. If trained on rebalanced data, rescale posteriors by population priors over training priors and renormalise.
  4. Set the threshold at c0/(c0+c1) for two classes, or assign each case to the class with minimum expected loss.
  5. Optionally set a reject threshold and route uncertain cases to human review, tracking the reject fraction.
  6. Compute expected cost on test data for each model and for the trivial rules, and choose the lowest.

Sources

  • Handbook of Data Mining (Ye, 2003)
  • Deep Learning: Foundations and Concepts (Bishop & Bishop, 2024)
  • Bayesian Analysis for the Social Sciences (Jackman, 2009)
  • Applied Predictive Modeling (Kuhn & Johnson, 2013)
  • Pattern Classification (Duda et al., 2001)
  • Pattern Recognition and Machine Learning (Bishop, 2006)

Also known as

  • Cost-sensitive decision threshold
  • Choose the loss first
  • Expected-cost threshold
  • Cost-based cut-off
  • Minimum-expected-loss decision rule
  • Loss matrix with reject option
  • Class-prior compensation

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning