Programmatic Labelling with Weak Supervision

Description

Encode experts' labelling heuristics as functions, apply them to unlabelled data and combine their noisy, conflicting votes into training labels, checked against a small hand-labelled sample.

Weak supervision replaces most hand labelling with labelling functions that encode heuristics: keywords, regular expressions, database lookups or other models' outputs. Each function labels only what it recognises and may be wrong, and functions can conflict, so their outputs are combined, denoised and reweighted into most-likely labels by a label model. A small hand-labelled sample is recommended to estimate each function's accuracy and to inspire better functions. Advantages include cost, since expertise is captured once and can be versioned and reused; privacy, since only a cleared subsample needs human inspection; and adaptivity, since functions can be reapplied when data or requirements change. A model trained on the programmatic labels can generalise to cases no function covers. For research teams, it offers a way to scale qualitative coding schemes to large volumes of feedback with explicit, auditable rules.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Days

Steps

  1. Hand-label a small sample to learn the patterns and to estimate heuristic accuracy.
  2. Write labelling functions that encode expert heuristics (keywords, regular expressions, lookups, other models' outputs).
  3. Apply all functions to the unlabelled data.
  4. Combine, denoise and reweight the functions' conflicting outputs into most-likely labels with a label model.
  5. Check label quality against the hand-labelled sample and revise weak functions.
  6. Optionally train a model on the labels to generalise beyond function coverage, and reapply the functions when data or requirements change.

Sources

  • Designing Machine Learning Systems (Huyen, 2022)

Also known as

  • Weak supervision
  • Labelling functions
  • Data programming
  • Snorkel

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning