Active Learning for Annotation

Description

Let a model choose the unlabelled items it is least sure about, have people label those, retrain and repeat, so each costly label improves the model most.

Labelling data is expensive, and random samples waste effort on easy cases. In active learning a model trained on a small labelled set scores the unlabelled pool by how informative each item would be: uncertainty sampling picks items whose predicted class has the lowest probability or lie near the decision boundary; query-by-committee picks items on which models trained with different settings or data slices disagree; other heuristics estimate expected loss reduction. The chosen items go to annotators, labels are added and the model is retrained, repeating until performance is sufficient. Candidates can come from a stored pool or a live stream, which lets models adapt to changing data. Settles' toy example reached 90 per cent accuracy with 30 actively chosen labels against 70 per cent with 30 random ones. For UX research, it reduces the effort of coding large qualitative corpora with a classifier.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Hand-label a small seed set and train an initial model.
  2. Score unlabelled items by uncertainty, committee disagreement or expected loss reduction.
  3. Send the highest-scoring items to annotators, with clear coding guidance.
  4. Add the new labels and retrain the model.
  5. Track performance on a fixed, randomly sampled validation set.
  6. Repeat until performance plateaus or the labelling budget is spent.

Sources

  • Designing Machine Learning Systems (Huyen, 2022)
  • Pattern Classification (Duda et al., 2001)

Also known as

  • Learning with queries
  • Query learning
  • Uncertainty sampling
  • Query-by-committee

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning