Design Bakeoff

Description

Several teams build semi-functional alternatives; participants use them in a counterbalanced design and the designs are compared on issues, ratings and forced-choice preference.

Early in a project, rival design directions are best settled with evidence. In a design bakeoff, two or more teams build semi-functional prototypes of alternatives for the same tasks. With a large sample, a between-subjects design can compare them directly. More often each participant works mainly with one primary design, counterbalanced across participants, and is then shown the other designs briefly and asked for a forced-choice preference and ratings. Results compare the number and severity of issues found per design together with preference shares and ratings. Because the primary design receives far more use, conclusions about briefly seen designs must be cautious, and small samples support issue comparisons better than statistical preference claims. A common outcome is merging the best elements rather than choosing one design wholesale.

Attributes

Loss type
Renamed
Stage
Design
Data
Mixed
Effort
Weeks

Steps

  1. Have two or more teams build semi-functional prototypes of alternative designs for the same core tasks.
  2. Choose between-subjects comparison if the sample is large; otherwise assign each participant one primary design, counterbalancing which design is primary.
  3. Run the core tasks on the primary design, recording issues and their severity.
  4. Show the other designs briefly and ask for a forced-choice preference and ratings of each.
  5. Compare issue frequency and severity by design, preference counts and mean ratings, with confidence intervals where samples allow.
  6. Decide which design, or which elements of each, to carry forward.

Sources

  • Measuring the User Experience (Albert & Tullis, 2022)
  • Handbook of Usability Testing (Rubin & Chisnell, 2008)

Also known as

  • Comparing alternative designs
  • Design shoot-out
  • Comparative usability test

Categories

  • User Research
  • Usability & Evaluation