Description
Measure whether an experimental manipulation actually moved the intended construct, and not merely related ones, ideally in pilots or with its position counterbalanced.
An experiment that varies 'trust cues' or 'urgency' assumes the manipulation changed users' perceived trust or urgency. A manipulation check tests that assumption, supporting internal and construct validity. Only about a third of published studies report one, usually a single self-report item. Good practice measures the target construct plus related and unrelated constructs for convergent and divergent evidence, uses more than one kind of measure (self-report, open text, behaviour), and sets failure criteria in advance. A failed check may reflect a poor measure, an insensitive check or a failed manipulation. Because the check is itself an event that can sharpen or dampen the state it measures and change later behaviour, it is best placed in pilots, or its position counterbalanced in the main study with order effects tested. Results should be analysed as strength, not just pass or fail.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Latent
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Define the construct the manipulation should change, plus related constructs it should not change and unrelated ones as controls.
- Choose measures: self-report items, open-text responses or behavioural indicators.
- Set failure criteria in advance, including what you will do if the check fails.
- Run the check in a pilot study, or counterbalance its position (before or after the outcome) in the main study and test for order effects.
- Analyse the strength of the effect on the target construct and the absence of effects on divergent constructs, not just pass or fail.
- Report the check fully, including failures.
Sources
- Cambridge Handbook of Research Methods (Nichols & Edlund, 2023)
- Chatbot Research and Design: CONVERSATIONS 2022 (Følstad et al., 2023)
Also known as
- Construct validity check of a manipulation
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference