Expectation Ratings (Expected vs Experienced Difficulty)

Description

Compares how hard users expect each task to be with how hard it turns out, mapping tasks into fix-it-fast, promote-it, big-opportunity and leave-alone quadrants.

Before doing any tasks, participants rate how difficult they expect each one to be. After each task, they rate how difficult it actually was, using the same 7-point scale. Plotting expectation against experience separates four groups: tasks expected to be easy that turned out hard ("fix it fast", the main source of dissatisfaction); tasks expected to be hard that were easy ("promote it"); tasks hard before and after ("big opportunity"); and tasks easy before and after ("don't touch it"). The method adds context a single post-task rating lacks, because the same difficulty score means something different if users expected it. Sauro reports that users predict task-level usability fairly well, which makes the gaps meaningful.

Attributes

Loss type
Latent
Stage
Evaluation
Effort
Hours

Steps

  1. List the study tasks and show them all to participants before testing.
  2. Collect an expectation rating for each task on a 7-point difficulty scale.
  3. Run the tasks, collecting the experience rating after each one on the same scale.
  4. Average both ratings per task and plot expectation on one axis and experience on the other.
  5. Prioritise the quadrant where tasks were expected to be easy but were hard.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Measuring the User Experience (Tullis & Albert)

Also known as

  • Expectation ratings
  • ER
  • Expectation-experience quadrant

Categories

  • User Research
  • Usability & Evaluation