Top-Box Analysis of Rating Scales

Description

Converts rating responses to the share choosing the top one or two options, then analyses that share as a proportion.

Dashboards often report "percent who agree" (top-two-box on a 5-point scale) rather than means. To test a claim such as "at least 75% agree", convert 4s and 5s to 1 and everything else to 0, then use the binomial methods for completion rates: a mid-p test against the benchmark and an adjusted-Wald interval. Sauro and Lewis show the trade-off clearly: converting discards intensity, so the same data that gives a 72.5% chance of meeting the goal as top-box gives 81.4% when analysed as a mean with a t-test. Top-box is easy to communicate but needs larger samples to detect changes, so analyse raw means where you can and report top-box where dashboards require it.

Attributes

Loss type
Active
Effort
Hours

Steps

  1. Decide the top-box rule before analysis (for example 4 or 5 on a 5-point scale).
  2. Convert each response to 1 or 0.
  3. Compute the proportion with an adjusted-Wald interval.
  4. Test against any benchmark with the mid-p binomial test for small samples.
  5. Also report the mean and its interval so intensity is not lost.

Sources

  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • Top-box
  • Top-two-box
  • Net top-minus-bottom box

Categories

  • Analytics & Behaviour
  • Statistics & Modelling