Description
Participants view two or more design versions, choose the one they prefer and explain why, with presentation order randomised to control order bias.
A preference test asks which of two (or several) versions a participant prefers and why. It measures stated preference, not behaviour, so it complements rather than replaces task-based testing or live split tests. Versions should differ in only one or two elements so the reasons are interpretable. Order or labelling must be randomised per participant, and counts normalised for label swaps, because first-shown and A-labelled options attract choices. The open 'why' answers are thematically analysed and often reveal novelty or familiarity effects. Armitage's graded-embellishment variant shows one design in several versions with styling added cumulatively on identical layout and data (for example bars and sparklines, then frames and shading, then gradients or traffic-light backgrounds), locating the point where decoration stops helping. A near 50/50 split signals that neither option is clearly better.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Active
- Stage
- Design, Evaluation
- Data
- Mixed
- Effort
- Days
Steps
- Prepare versions that differ in one or two elements, or a graded series adding treatments cumulatively on identical layout and content.
- Choose delivery: in person, or an online survey with static images or prototype links and instructions on a separate intro page.
- Randomise which version appears first or is labelled A for each participant.
- Have participants use the target device, ideally their own, and ask one either/or (or 'most effective') question followed by an open 'why?'.
- Count preferences after normalising for label swaps and test the split against chance; a near 50/50 split means return to an earlier design stage.
- Thematically analyse the reasons, watching for novelty and familiarity effects.
- Confirm important choices with behavioural data such as task testing or a live A/B test.
Sources
- UX Research Methods for Media and Communication Studies (Cirucci & Pruchniewska, 2022)
- Bringing Numbers to Life (Armitage, 2015)
- Modern Awards User Experience Research Report (Wallis Market and Social Research, 2021)
- Survey Analysis (Sauro & Lewis, 2021)
- Benchmarking the User Experience (Sauro, 2018)
- Customer Analytics for Dummies (Sauro, 2015)
Also known as
- A/B preference test
- Side-by-side preference test
- Graded-embellishment preference test
Categories
- User Research
- Usability & Evaluation