Description
Decides whether a product beats a specific target, such as 80% completion or a SUS of 68, rather than just reporting a score.
Teams often set goals like "at least 75% of users complete the task" or "SUS above 68" and then eyeball whether the sample result clears the bar. A one-sample test makes the decision explicit. For completion rates with small samples, the exact (mid-p) binomial test gives the probability of seeing this many successes if the true rate were only at the benchmark. For larger samples a normal approximation works. For continuous measures such as SUS, SEQ or log task times, a one-sample t-test compares the mean to the target. Because the claim is directional (better than the goal), one-tailed tests are appropriate, and the same logic can be read off a confidence interval: the goal is met if it lies below the lower bound.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- State the benchmark and direction before collecting data (for example, completion > 70%).
- For proportions with n under about 30, run an exact or mid-p binomial test of x successes in n trials against the benchmark p.
- For larger samples, use the z test with the normal approximation.
- For rating scales and log-transformed times, run a one-sample t-test of the mean against the benchmark value.
- Report the one-tailed p-value alongside the confidence interval and state plainly whether the goal was met.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- How to Conduct a Quantitative Usability Test (Sauro)
- Benchmarking the User Experience (Sauro)
- Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
Also known as
- Did we meet our goal?
- One-sample t-test against target
- Exact binomial test against target
Categories
- Analytics & Behaviour
- Statistics & Modelling