Description
Reports the geometric mean of skewed measures such as task times as the best estimate of the typical value, with confidence intervals derived on the log scale.
For positively skewed data, the arithmetic mean overstates typical performance and the sample median is noisy and biased for small samples. Sauro and Lewis compared estimators on large usability datasets and found the geometric mean (the exponential of the mean of log values) had less error and bias in estimating the population median for sample sizes under about 25, while the sample median performed better above that. The geometric mean also makes multiplicative comparisons natural, for example one design taking 0.8 times as long. Rigour comes from computing confidence intervals on the log scale and back-transforming them, explaining clearly that the result estimates the median, and using it consistently across benchmark rounds.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Latent
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Confirm the measure is strictly positive and positively skewed, for example task times, page load times or ratios.
- Compute the natural log of each value and calculate the mean and standard deviation of the logs.
- Exponentiate the log mean to obtain the geometric mean.
- Compute a 95% confidence interval on the log scale (log mean ± t × SD/√n) and exponentiate the limits to obtain an asymmetric interval.
- For samples of about 25 or more, also report the sample median, and choose a consistent primary estimate for benchmarking.
- Report results with wording such as 'typical completion time' and include the arithmetic mean only if total time or cost is the decision-relevant quantity.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Statistics for HCI (Dix)
- Practical Statistics for Data Scientists (Bruce)
- The Inquiry Series, Ch 39 — The Distribution of Human Performance — Why Averages Lie
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- How to Conduct a Quantitative Usability Test (Sauro)
Also known as
- Geometric mean task time
- Back-transformed log mean
- Median estimation for small samples
Categories
- Analytics & Behaviour
- Statistics & Modelling