Supervised text categorisation

Description

Train classifiers on pre-tagged documents to assign categories to new text, choosing each category's threshold from ranked precision-recall trade-offs.

Feldman contrasts hand-written classification rules, which cost person-years of knowledge engineering, with learning from pre-tagged documents. Classifiers may give hard yes or no answers or ranked scores. Methods include naive Bayes, k-nearest neighbours (k of 30-45 works well and larger k does little harm), rule learning with a loss ratio trading precision for recall, and support vector machines. One classifier is trained per category; held-out documents are scored and ranked, and a visual tool lets the user pick each category's threshold from its precision-recall curve. In the example, ten classifiers were tested on 476 documents, and a threshold of 0.829 gave 97 per cent precision and 98 per cent recall for one category. In UX the same procedure tags support tickets, reviews and open-ended survey answers at scale, now usually under labels such as automated coding or LLM classification.

Attributes

Loss type
Renamed
Data
Mixed
Effort
Weeks

Steps

  1. Define the category scheme and tag a training set, ideally with two coders and an agreement check.
  2. Train one classifier per category (for example naive Bayes, kNN, SVM or a fine-tuned language model).
  3. Score held-out documents and rank them per category.
  4. Plot precision and recall against threshold and pick each cut-off by the cost of errors.
  5. Apply to new text and route borderline cases to human review.
  6. Re-check performance periodically against freshly tagged samples as language drifts.

Sources

  • Handbook of Data Mining (Ye, 2003)

Also known as

  • Supervised text categorisation with ranked-threshold tuning
  • Text classification
  • Document categorisation
  • Automatic tagging

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning