Benchmarking

Description

Measure task performance or reported experience to compare an interface with earlier versions or competitors.

Measure how well people complete tasks so you can compare an interface with earlier versions or competitors. Run a moderated or unmoderated summative usability evaluation and collect performance measures such as completion rates and task times. Benchmarking can also compare reported attitudes, such as trust and appearance, through retrospective surveys. Use the comparison to assess the user experience across products or versions.

Attributes

Loss type
Active
Effort
Weeks

Steps

  1. Select 5 to 10 key tasks that represent core value and define success criteria.
  2. Choose metrics: task success, time, errors, task ease (SEQ) and a validated questionnaire such as SUS.
  3. Calculate the sample size needed for target precision and recruit consistently across rounds.
  4. Run the study with a fixed protocol, usually unmoderated for scale.
  5. Analyse metrics with confidence intervals and compare against baseline or competitors using appropriate tests.
  6. Create a scorecard and repeat at regular intervals, such as every release or year.

Sources

  • Benchmarking the User Experience
  • Quantifying the User Experience (Sauro & Lewis)
  • How to Conduct a Quantitative Usability Test (Sauro)
  • Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)

Videos

Also known as

  • UX benchmarking
  • usability benchmarking
  • Competitive usability benchmarking
  • Summative evaluation

Categories

  • User Research
  • Usability & Evaluation