Threats-to-Validity Audit (Shadish, Cook and Campbell)

Description

Systematically checks a study against named threats to statistical conclusion, internal, construct and external validity, and designs them out before data collection.

Campbell's tradition catalogues specific threats: for internal validity, history, maturation, testing, instrumentation, regression to the mean, selection, attrition and their interactions; for construct validity, mono-operation bias, hypothesis guessing, experimenter expectancies, novelty effects; for statistical conclusion validity, low power, fishing, unreliable measures; for external validity, interactions of treatment with units, settings and outcomes. The audit names each threat, assesses its plausibility and specifies design features that rule it out (e.g. a comparison group, multiple pretests, a non-equivalent dependent variable). This structured scepticism is rarely applied to UX studies, where novelty effects and regression to the mean routinely masquerade as findings.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Describe the study design in Campbell notation (O and X sequences, R for randomisation, dashed lines for non-equivalent groups).
  2. Go through each named threat for the four validity types and rate plausibility given the context.
  3. For plausible threats, add design elements: control groups, extra pretests, holdouts, blinded assessment, multiple measures.
  4. Plan analyses that test specific threats (e.g. checking regression to the mean when sampling on extreme scores).
  5. Document residual threats explicitly in the report with their likely direction of bias.
  6. Have a colleague red-team the audit before data collection.

Sources

  • Shadish, Cook & Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002)
  • Campbell & Stanley, Experimental and Quasi-Experimental Designs for Research (1966)

Also known as

  • Validity threats checklist
  • Campbell's validity typology
  • Internal/external/construct/statistical conclusion validity

Categories

  • Research Operations
  • Research Design & Rigor