Testing Against a Benchmark (One-Sample Tests)

Description

Decides whether a product beats a specific target, such as 80% completion or a SUS of 68, rather than just reporting a score.

Teams often set goals like "at least 75% of users complete the task" or "SUS above 68" and then eyeball whether the sample result clears the bar. A one-sample test makes the decision explicit. For completion rates with small samples, the exact (mid-p) binomial test gives the probability of seeing this many successes if the true rate were only at the benchmark. For larger samples a normal approximation works. For continuous measures such as SUS, SEQ or log task times, a one-sample t-test compares the mean to the target. Because the claim is directional (better than the goal), one-tailed tests are appropriate, and the same logic can be read off a confidence interval: the goal is met if it lies below the lower bound.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. State the benchmark and direction before collecting data (for example, completion > 70%).
  2. For proportions with n under about 30, run an exact or mid-p binomial test of x successes in n trials against the benchmark p.
  3. For larger samples, use the z test with the normal approximation.
  4. For rating scales and log-transformed times, run a one-sample t-test of the mean against the benchmark value.
  5. Report the one-tailed p-value alongside the confidence interval and state plainly whether the goal was met.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • How to Conduct a Quantitative Usability Test (Sauro)
  • Benchmarking the User Experience (Sauro)
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)

Also known as

  • Did we meet our goal?
  • One-sample t-test against target
  • Exact binomial test against target

Categories

  • Analytics & Behaviour
  • Statistics & Modelling