Description
Systematically checks a study against named threats to statistical conclusion, internal, construct and external validity, and designs them out before data collection.
Campbell's tradition catalogues specific threats: for internal validity, history, maturation, testing, instrumentation, regression to the mean, selection, attrition and their interactions; for construct validity, mono-operation bias, hypothesis guessing, experimenter expectancies, novelty effects; for statistical conclusion validity, low power, fishing, unreliable measures; for external validity, interactions of treatment with units, settings and outcomes. The audit names each threat, assesses its plausibility and specifies design features that rule it out (e.g. a comparison group, multiple pretests, a non-equivalent dependent variable). This structured scepticism is rarely applied to UX studies, where novelty effects and regression to the mean routinely masquerade as findings.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Evaluation, Strategy
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Describe the study design in Campbell notation (O and X sequences, R for randomisation, dashed lines for non-equivalent groups).
- Go through each named threat for the four validity types and rate plausibility given the context.
- For plausible threats, add design elements: control groups, extra pretests, holdouts, blinded assessment, multiple measures.
- Plan analyses that test specific threats (e.g. checking regression to the mean when sampling on extreme scores).
- Document residual threats explicitly in the report with their likely direction of bias.
- Have a colleague red-team the audit before data collection.
Sources
- Shadish, Cook & Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002)
- Campbell & Stanley, Experimental and Quasi-Experimental Designs for Research (1966)
Also known as
- Validity threats checklist
- Campbell's validity typology
- Internal/external/construct/statistical conclusion validity
Categories
- Research Operations
- Research Design & Rigor