Description
Tests whether three or more designs differ on a continuous measure, and whether two factors interact.
When comparing more than two designs, running every pairwise t-test inflates false positives. ANOVA tests all groups at once by comparing variation between groups to variation within them (the F-statistic). Two-way ANOVA adds a second factor, such as device or experience level, and tests their interaction: does the new design help novices more than experts? Significant results are followed by planned contrasts or corrected post-hoc comparisons. Bruce and Bruce also show a resampling (permutation) version that avoids distributional assumptions.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Define factors and levels (for example, design A/B/C by device desktop/mobile).
- Log-transform task times; check group variances are not wildly different.
- Run one-way or two-way ANOVA and read the F tests for main effects and interactions.
- Follow significant effects with corrected pairwise comparisons (Tukey, Holm, Bonferroni).
- Report effect sizes (partial eta squared) and plots of group means with intervals.
Sources
- Quantifying the User Experience (Sauro & Lewis)
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Benchmarking the User Experience (Sauro)
- Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
- Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)
Also known as
- Analysis of variance
- F-test
- Factorial ANOVA
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference