Latin Square and Williams Counterbalancing

Description

Orders conditions in within-subjects studies so each appears equally often in each position, and with Williams designs, follows each other equally often.

In within-subject usability studies, order and carryover effects (practice, fatigue, learning) confound condition comparisons. A Latin square assigns condition orders so each condition appears once in each position across a set of participants. A Williams (balanced Latin square) design additionally ensures each condition immediately follows every other equally often, balancing first-order carryover; for an odd number of conditions, two squares are needed. Analysis should include order and participant as factors. Many UX studies 'randomise' with a handful of participants, which leaves imbalance; or they simply fix order. Proper counterbalancing is cheap rigour.

Attributes

Loss type
Vestigial
Stage
Evaluation
Effort
Hours

Steps

  1. List the conditions (designs, tasks) to be compared within subjects.
  2. Generate a Williams design for the number of conditions (one square for even n, two for odd n).
  3. Recruit participants in multiples of the number of sequences so the design is complete.
  4. Assign participants to sequences at random.
  5. Analyse with mixed models including condition, period and (optionally) carryover terms, with participant as random effect.
  6. Report order effects and check whether any condition shows asymmetric transfer.

Sources

  • Williams, Experimental designs balanced for the estimation of residual effects of treatments (1949), Australian Journal of Scientific Research
  • MacKenzie, Human-Computer Interaction: An Empirical Research Perspective (2013)
  • How to Conduct a Quantitative Usability Test (Sauro)
  • Benchmarking the User Experience (Sauro)

Also known as

  • Latin square design
  • Williams design
  • Balanced Latin square
  • Counterbalancing

Categories

  • Research Operations
  • Research Design & Rigor