Description
Identifies and controls cues that let participants guess what researchers want and researcher behaviours that shift responses, reducing positive bias in usability findings.
Demand characteristics are the cues in a study that tell participants what the hypothesis is, leading them to behave as 'good subjects'. Experimenter effects are the ways researchers' expectations and behaviour influence responses. In UX both inflate positive findings: participants know a designer may be present, praise prototypes, persist longer than they would at home, and rate satisfaction higher when the moderator built the design. Dumas and Loring and others have documented these moderator effects. Control techniques include neutral, scripted instructions; moderators who are not the designers; disclosing that the team wants to find problems; testing competitor or multiple versions so the 'favoured' one is unclear; behavioural rather than attitudinal measures; post-session suspicion probes; and anonymous ratings collected away from the moderator. These are standard in experimental psychology but rare in UX.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Evaluation, Design
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Control strategies:
- Use cover stories or task framing that doesn't reveal the study's hypothesis ('we're testing the navigation' not 'we're testing whether the new design is better').
- (2) Remote unmoderated testing eliminates experimenter presence.
- (3) Counterbalance: evaluator knowledge of design condition should be blind.
- (4) Measure suspicion: post-session ask 'what do you think we were testing?' Participants who correctly identify the hypothesis are excluded or analyzed separately.
Sources
- Research Methods in Psychology — Schweigert
- Orne, On the social psychology of the psychological experiment (American Psychologist, 1962)
- Rosenthal, Experimenter Effects in Behavioral Research (1966)
- Moderating Usability Tests (Joseph Dumas & Beth Loring)
- Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
- Garcia-Ruiz, M. (ed.), Games User Research: A Case Study Approach (2016)
Also known as
- Reactivity control
- Experimenter bias control
- Social desirability control in usability tests
Categories
- Research Operations
- Research Design & Rigor