Audit (correspondence) study

Description

A field experiment that submits matched profiles differing in one signal to real decision makers or systems and compares responses, isolating discrimination by that signal.

Audit studies measure discrimination in real decisions rather than self-report. Researchers build sets of applications, accounts or profiles that are identical in every relevant respect except one signal, such as a name implying ethnicity, an age or a postcode, and submit them to the same real opportunities in randomised order. Differences in callbacks, prices, rankings or moderation decisions can then be attributed to the varied signal. In labour economics, matched CVs answering the same job adverts revealed large callback gaps by name. The design transfers directly to algorithm audits: sock-puppet accounts let researchers test whether recommendation, pricing, ad-delivery or moderation systems treat equivalent users differently. Rigour comes from tight matching, randomised submission order and enough pairs to estimate differential response rates.

Attributes

Loss type
Unmigrated
Effort
Weeks

Steps

  1. Define the attribute to test (for example name-implied ethnicity, gender or location) and the outcome (callback, price, rank, approval).
  2. Build sets of profiles identical on all relevant characteristics except the test attribute, and pre-test that the signal is perceived as intended.
  3. Submit each set to the same real opportunities or system queries, randomising which version goes first.
  4. Record responses systematically (callbacks, quoted prices, ranking positions, moderation outcomes).
  5. Compare response rates between versions with paired analysis and report the differential with its uncertainty.
  6. Review ethics: minimise burden on real decision makers and consider whether the signal carries more than one meaning.

Sources

  • Handbook of Child Psychology Vol 3 (Lerner & Lamb, 2015)

Also known as

  • Audit (correspondence) study with matched profiles
  • Correspondence test
  • Paired testing
  • Sock-puppet audit

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference