Description
Measures how well a classifier, human coder or AI feature sorts things correctly, and what kinds of mistakes it makes.
Accuracy alone hides the errors that matter. A confusion matrix counts true and false positives and negatives. Precision says how many flagged items were right; recall says how many of the real items were caught; specificity covers the negatives. The ROC curve shows the trade-off across thresholds and AUC summarises it. Lift shows how much better than random the top-ranked cases are. For UX researchers these are essential when evaluating AI features (does the spam filter or recommendation help users?), auto-coding tools, and screening questions in recruitment.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Unmigrated
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Collect ground-truth labels for a test set.
- Build the confusion matrix at the chosen threshold.
- Compute precision, recall, specificity and F1.
- Plot ROC and precision-recall curves and compute AUC.
- Choose thresholds based on the user cost of each error type, not on accuracy.
Sources
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
Also known as
- Confusion matrix
- ROC/AUC analysis
- Precision-recall
- Lift chart
Categories
- Analytics & Behaviour
- Computational & Machine Learning