Record Linkage

Description

Identifying the same person or unit across a survey and other datasets with error-tolerant deterministic or probabilistic matching, including privacy-preserving encrypted linkage.

Linking survey answers to administrative, account or behavioural records multiplies their value, but identifiers are messy. Record linkage first obtains consent, then preprocesses identifiers (encodings, name variants, date formats, addresses, missing codes). Because comparing every pair is infeasible, pairs are blocked on several keys in turn. Similarity is computed per field (Jaro-Winkler for names), and the Fellegi-Sunter framework turns agreement patterns into match weights, with an upper threshold for links, a lower one for non-links and clerical review between them. Privacy-preserving variants use a trusted third party or encode identifiers in Bloom filters so matching occurs without exposing raw data. Practical warnings: consent to linkage ranges from 24 to 89 per cent across surveys, non-consent and missed matches are not random, and preprocessing and approvals take far longer than the linking itself.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Obtain informed consent to link and agree governance and approvals.
  2. Preprocess identifiers: standardise encodings, names, dates, addresses and missing codes.
  3. Block candidate pairs on several keys in turn to limit comparisons.
  4. Compute field similarities (for example Jaro-Winkler for names, exact for dates).
  5. Classify pairs with probabilistic weights and upper and lower thresholds; send the grey zone to clerical review.
  6. For sensitive data, link through a trusted third party or with encrypted Bloom filter encodings.
  7. Report match rates, consent rates and how linked cases differ from unlinked ones.

Sources

  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)

Also known as

  • Entity resolution
  • Probabilistic record linkage
  • Fellegi-Sunter linkage
  • Privacy-preserving record linkage
  • Bloom filter linkage

Categories

  • Analytics & Behaviour
  • Statistics & Modelling