Negative Control Analysis

Description

Tests an analysis on an outcome or exposure that should show no effect; any 'effect' found reveals confounding or bias in the method.

A negative control outcome is one that cannot plausibly be affected by the treatment but shares the same confounders; a negative control exposure is one that cannot affect the outcome but shares confounding with the real exposure. If the analysis finds an effect on a negative control, the method is biased. Examples in UX: a new search feature should not change the rate of password resets; users exposed to a campaign last month should not show effects before it started. Negative controls are cheap, powerful falsification checks for observational analyses and also for A/B tests with suspected sample ratio mismatch or instrumentation issues.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Identify the main causal analysis and its likely sources of bias (self-selection, seasonality, instrumentation).
  2. Choose negative control outcomes affected by the same biases but not by the treatment.
  3. Choose negative control exposures or placebo times (e.g. a fake launch date) where appropriate.
  4. Run the identical analysis pipeline on the negative controls.
  5. Interpret non-null results as evidence of bias and adjust or abandon the method.
  6. Report negative control results alongside main estimates.

Sources

  • Lipsitch, Tchetgen Tchetgen & Cohen, Negative controls: a tool for detecting confounding and bias in observational studies (2010), Epidemiology
  • Arnold et al., Negative control outcomes: a tool to detect bias (2016), International Journal of Epidemiology

Also known as

  • Negative control outcomes
  • Negative control exposures
  • Falsification tests
  • Placebo tests

Categories

  • Epidemiology & Population Health