Classifier Evaluation (Confusion Matrix, Precision, Recall, ROC, Lift)

Description

Measures how well a classifier, human coder or AI feature sorts things correctly, and what kinds of mistakes it makes.

Accuracy alone hides the errors that matter. A confusion matrix counts true and false positives and negatives. Precision says how many flagged items were right; recall says how many of the real items were caught; specificity covers the negatives. The ROC curve shows the trade-off across thresholds and AUC summarises it. Lift shows how much better than random the top-ranked cases are. For UX researchers these are essential when evaluating AI features (does the spam filter or recommendation help users?), auto-coding tools, and screening questions in recruitment.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Collect ground-truth labels for a test set.
  2. Build the confusion matrix at the chosen threshold.
  3. Compute precision, recall, specificity and F1.
  4. Plot ROC and precision-recall curves and compute AUC.
  5. Choose thresholds based on the user cost of each error type, not on accuracy.

Sources

  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)

Also known as

  • Confusion matrix
  • ROC/AUC analysis
  • Precision-recall
  • Lift chart

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning