Description
A decision procedure for choosing within-subjects, between-subjects or mixed designs in UX studies, weighing error variance, carryover effects and sample size.
In a within-subjects design every participant experiences all conditions, which removes between-person variability and typically needs far fewer participants, but introduces order, practice, fatigue and carryover effects that must be controlled through counterbalancing such as Latin squares. In a between-subjects design each participant sees one condition, avoiding carryover but requiring larger samples and random assignment. Many UX comparisons are run without considering this choice, for example letting participants try both prototypes in the same order, or running unmoderated between-subjects tests underpowered by a factor of three. The method makes the choice explicit: assess whether exposure to one condition changes the response to another, estimate required samples for each design from expected effect size and variability, and specify counterbalancing or randomisation.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Design selection decision tree:
- Is carryover possible (learning, fatigue, practice effects)? YES → between-subjects or counterbalanced within-subjects. NO → within-subjects (60% fewer participants needed).
- (2) If within-subjects: counterbalance condition order using Latin square.
- (3) For preference studies: always within-subjects (between-subjects preference studies are severely underpowered).
- (4) Compute required N using G*Power for the chosen design.
Sources
- Research Methods in Psychology — Wendy A. Schweigert
- Research Methods in Psychology: A Handbook (Wendy Schweigert)
- Quantifying the User Experience (Jeff Sauro & James Lewis)
Also known as
- Repeated measures versus independent groups
- Experimental design selection
Categories
- Research Operations
- Research Design & Rigor