Description
Train classifiers on pre-tagged documents to assign categories to new text, choosing each category's threshold from ranked precision-recall trade-offs.
Feldman contrasts hand-written classification rules, which cost person-years of knowledge engineering, with learning from pre-tagged documents. Classifiers may give hard yes or no answers or ranked scores. Methods include naive Bayes, k-nearest neighbours (k of 30-45 works well and larger k does little harm), rule learning with a loss ratio trading precision for recall, and support vector machines. One classifier is trained per category; held-out documents are scored and ranked, and a visual tool lets the user pick each category's threshold from its precision-recall curve. In the example, ten classifiers were tested on 476 documents, and a threshold of 0.829 gave 97 per cent precision and 98 per cent recall for one category. In UX the same procedure tags support tickets, reviews and open-ended survey answers at scale, now usually under labels such as automated coding or LLM classification.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Renamed
- Stage
- Post-launch, Discovery
- Data
- Mixed
- Effort
- Weeks
Steps
- Define the category scheme and tag a training set, ideally with two coders and an agreement check.
- Train one classifier per category (for example naive Bayes, kNN, SVM or a fine-tuned language model).
- Score held-out documents and rank them per category.
- Plot precision and recall against threshold and pick each cut-off by the cost of errors.
- Apply to new text and route borderline cases to human review.
- Re-check performance periodically against freshly tagged samples as language drifts.
Sources
- Handbook of Data Mining (Ye, 2003)
Also known as
- Supervised text categorisation with ranked-threshold tuning
- Text classification
- Document categorisation
- Automatic tagging
Categories
- Analytics & Behaviour
- Computational & Machine Learning