Awareness-usefulness gap analysis

Description

Asks whether users knew of each feature, lets them try it, then rates its usefulness; features with high usefulness but low awareness need promotion or more prominent design.

The method needs two metrics per feature. Participants first answer a yes/no question on whether they were aware of each feature, asked before the feature is shown so the question does not create awareness. They then explore the feature for a couple of minutes and rate its usefulness on a 1-5 scale. Ratings are converted to top-2-box percentages and plotted beside the awareness percentages for each feature. Features with the largest gap between usefulness and awareness are useful but little known, and are candidates for better promotion, onboarding or more prominent placement. Features with high awareness and low usefulness are candidates for redesign or removal. The analysis is simple, but ordering the questions correctly and giving every participant the same brief exposure keep it honest.

Attributes

Loss type
Active
Effort
Days

Steps

  1. List the features of interest, typically 5-15.
  2. Before showing any feature, ask 'Were you aware of this feature?' (yes/no) for each.
  3. Let the participant explore each feature for a couple of minutes with a consistent prompt.
  4. Ask for a usefulness rating on a 1-5 scale after each exploration.
  5. Compute percentage aware and top-2-box percentage useful per feature, with confidence intervals.
  6. Plot the two percentages side by side, rank features by gap, and flag high-usefulness, low-awareness features.

Sources

  • Measuring the User Experience (Albert & Tullis, 2022)

Also known as

  • Usefulness-awareness gap
  • Feature awareness gap

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics