Description
Tests whether average ratings or task times differ between two designs, for the same or different participants.
The t-test is the workhorse for comparing two means: SUS scores, SEQ ratings, log task times. The paired version works on each participant's difference score in within-subjects designs and gains power by removing between-person variation. The two-sample version, preferably the Welch form that does not assume equal variances, compares independent groups. Sauro and Lewis show the t-test is robust for rating scales and works at small sample sizes, which settles a long debate about whether ordinal scale data can be analysed this way. Reporting the confidence interval around the difference matters more than the p-value.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Goal
- Compare designs
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Decide whether the design is within-subjects (paired) or between-subjects (two-sample).
- Log-transform task times first; leave rating scales as they are.
- For paired data, compute each person's difference and run a one-sample t-test on the differences against zero.
- For independent groups, run Welch's t-test using each group's own variance and adjusted degrees of freedom.
- Report the mean difference, its confidence interval, t, df and p.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Benchmarking the User Experience (Sauro)
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
- Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)
Also known as
- Welch t-test
- Paired t-test
- Student t-test
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference