Competitive UX Benchmarking

Description

Measures your product and competitors on the same tasks and metrics so you know where you stand.

A competitive benchmark runs identical tasks and questionnaires on two or more products. Key design choices are within-subjects (each person uses every product, which is efficient and allows direct preference questions but introduces carryover) versus between-subjects (cleaner but needing more people), and the mixed design where each participant uses their current product plus one competitor. Sauro's ten best practices include counterbalancing order, controlling for prior experience (prior users always rate a product higher), choosing tasks every product supports, and adjusting for brand attitudes. Results are reported as task and study metrics with confidence intervals and significance markers.

Attributes

Loss type
Active
Effort
Weeks

Steps

  1. Pick competitors and tasks that every product supports equivalently.
  2. Choose within, between or mixed design based on sample, carryover and task length.
  3. Counterbalance product order and record prior experience with each brand.
  4. Collect task metrics, SUPR-Q or SUS, preference and brand attitude before and after.
  5. Analyse with appropriate within or between tests, control for prior experience (for example weighted t-tests), and report gaps with intervals.

Sources

  • Benchmarking the User Experience (Sauro)
  • Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
  • Garcia-Ruiz, M. (ed.), Games User Research: A Case Study Approach (2016)

Also known as

  • Competitive usability benchmark
  • Head-to-head benchmark

Categories

  • User Research
  • Usability & Evaluation