Description
Orders conditions in within-subjects studies so each appears equally often in each position, and with Williams designs, follows each other equally often.
In within-subject usability studies, order and carryover effects (practice, fatigue, learning) confound condition comparisons. A Latin square assigns condition orders so each condition appears once in each position across a set of participants. A Williams (balanced Latin square) design additionally ensures each condition immediately follows every other equally often, balancing first-order carryover; for an odd number of conditions, two squares are needed. Analysis should include order and participant as factors. Many UX studies 'randomise' with a handful of participants, which leaves imbalance; or they simply fix order. Proper counterbalancing is cheap rigour.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Vestigial
- Goal
- Compare designs
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- List the conditions (designs, tasks) to be compared within subjects.
- Generate a Williams design for the number of conditions (one square for even n, two for odd n).
- Recruit participants in multiples of the number of sequences so the design is complete.
- Assign participants to sequences at random.
- Analyse with mixed models including condition, period and (optionally) carryover terms, with participant as random effect.
- Report order effects and check whether any condition shows asymmetric transfer.
Sources
- Williams, Experimental designs balanced for the estimation of residual effects of treatments (1949), Australian Journal of Scientific Research
- MacKenzie, Human-Computer Interaction: An Empirical Research Perspective (2013)
- How to Conduct a Quantitative Usability Test (Sauro)
- Benchmarking the User Experience (Sauro)
Also known as
- Latin square design
- Williams design
- Balanced Latin square
- Counterbalancing
Categories
- Research Operations
- Research Design & Rigor