ANOVA for Multi-Design Comparisons (One-Way and Two-Way)

Description

Tests whether three or more designs differ on a continuous measure, and whether two factors interact.

When comparing more than two designs, running every pairwise t-test inflates false positives. ANOVA tests all groups at once by comparing variation between groups to variation within them (the F-statistic). Two-way ANOVA adds a second factor, such as device or experience level, and tests their interaction: does the new design help novices more than experts? Significant results are followed by planned contrasts or corrected post-hoc comparisons. Bruce and Bruce also show a resampling (permutation) version that avoids distributional assumptions.

Attributes

Loss type
Active
Stage
Evaluation
Effort
Hours

Steps

  1. Define factors and levels (for example, design A/B/C by device desktop/mobile).
  2. Log-transform task times; check group variances are not wildly different.
  3. Run one-way or two-way ANOVA and read the F tests for main effects and interactions.
  4. Follow significant effects with corrected pairwise comparisons (Tukey, Holm, Bonferroni).
  5. Report effect sizes (partial eta squared) and plots of group means with intervals.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Benchmarking the User Experience (Sauro)
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
  • Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)

Also known as

  • Analysis of variance
  • F-test
  • Factorial ANOVA

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference