Description
Tests an analysis on an outcome or exposure that should show no effect; any 'effect' found reveals confounding or bias in the method.
A negative control outcome is one that cannot plausibly be affected by the treatment but shares the same confounders; a negative control exposure is one that cannot affect the outcome but shares confounding with the real exposure. If the analysis finds an effect on a negative control, the method is biased. Examples in UX: a new search feature should not change the rate of password resets; users exposed to a campaign last month should not show effects before it started. Negative controls are cheap, powerful falsification checks for observational analyses and also for A/B tests with suspected sample ratio mismatch or instrumentation issues.
Attributes
- Discipline
- Epidemiology & Population Health
- Loss type
- Unmigrated
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Identify the main causal analysis and its likely sources of bias (self-selection, seasonality, instrumentation).
- Choose negative control outcomes affected by the same biases but not by the treatment.
- Choose negative control exposures or placebo times (e.g. a fake launch date) where appropriate.
- Run the identical analysis pipeline on the negative controls.
- Interpret non-null results as evidence of bias and adjust or abandon the method.
- Report negative control results alongside main estimates.
Sources
- Lipsitch, Tchetgen Tchetgen & Cohen, Negative controls: a tool for detecting confounding and bias in observational studies (2010), Epidemiology
- Arnold et al., Negative control outcomes: a tool to detect bias (2016), International Journal of Epidemiology
Also known as
- Negative control outcomes
- Negative control exposures
- Falsification tests
- Placebo tests
Categories
- Epidemiology & Population Health