Description
Chooses a statistical test from the research question, the design (between, within or split-plot), the number of factors and conditions, the outcome's data type and assumptions.
Test selection is a decision procedure rather than a test. The question type (comparison with a norm, relationship, prediction, two or more independent or related groups, interactions) is crossed with the design and the outcome's level of measurement. For continuous outcomes meeting parametric assumptions: independent-samples t for two between groups, one-way ANOVA for three or more, factorial ANOVA for several factors; paired t and repeated-measures ANOVA for within designs; split-plot ANOVA for mixed designs. When assumptions fail (independent errors, equal variances, normal errors, interval scale) the alternatives are Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis and Friedman. Nominal outcomes go to chi-square, McNemar for paired proportions, or logistic regression for prediction. The rigour lies in fixing the design and analysis before data collection, using one-tailed tests only for directional hypotheses, and remembering that nonparametric tests lose power and still carry assumptions.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Active
- Stage
- Evaluation
- Data
- Analytical / expert
- Effort
- Hours
Steps
- State the question type: difference from a norm, relationship, prediction, difference between independent groups, between related measures, or interaction.
- Identify the design (between-group, within-group or split-plot), the number of independent variables and the number of conditions for each.
- Identify the outcome's data type: continuous, ordinal or nominal.
- For continuous outcomes, check independence, homogeneity of variance and normality of residuals.
- Read off the parametric test or its nonparametric or categorical alternative from the decision chart, and pre-register the choice where possible.
- Lay out the data as the software requires and report the statistic with degrees of freedom, p value and an effect size with confidence interval.
Sources
- Research Methods in HCI (Lazar et al., 2017)
- Statistical Analysis in JASP (Goss-Sampson, 2020)
- Quantitative Psychological Research (Clark-Carter, 2004)
- Advanced Quantitative Data Analysis (Cramer, 2003)
- Data Types (Interaction Design Foundation, n.d.)
- Survey Analysis (Sauro & Lewis, 2021)
- Confidence Intervals (Sauro & Lewis, 2021)
- Quantitative Social Research Methods (Singh, 2007)
- Benchmarking the User Experience (Sauro, 2018)
- Human-Computer Interaction (Dix et al., 2004)
- Statistics (Witte & Witte, 2017)
- Intermediate Statistics for Dummies (Rumsey, 2007)
Also known as
- Significance test selection by experimental design
- Test selection by question and data type
- Table 4.3 test selection
- Choosing a test for comparing means
- Which test should I use
- Statistical test selection
Categories
- Analytics & Behaviour
- Statistics & Modelling