t-Tests for Comparing Designs (Paired and Two-Sample)

Description

Tests whether average ratings or task times differ between two designs, for the same or different participants.

The t-test is the workhorse for comparing two means: SUS scores, SEQ ratings, log task times. The paired version works on each participant's difference score in within-subjects designs and gains power by removing between-person variation. The two-sample version, preferably the Welch form that does not assume equal variances, compares independent groups. Sauro and Lewis show the t-test is robust for rating scales and works at small sample sizes, which settles a long debate about whether ordinal scale data can be analysed this way. Reporting the confidence interval around the difference matters more than the p-value.

Attributes

Loss type
Active
Stage
Evaluation
Effort
Hours

Steps

  1. Decide whether the design is within-subjects (paired) or between-subjects (two-sample).
  2. Log-transform task times first; leave rating scales as they are.
  3. For paired data, compute each person's difference and run a one-sample t-test on the differences against zero.
  4. For independent groups, run Welch's t-test using each group's own variance and adjusted degrees of freedom.
  5. Report the mean difference, its confidence interval, t, df and p.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Benchmarking the User Experience (Sauro)
  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
  • Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)

Also known as

  • Welch t-test
  • Paired t-test
  • Student t-test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference