Statistical test selection by design

Description

Chooses a statistical test from the research question, the design (between, within or split-plot), the number of factors and conditions, the outcome's data type and assumptions.

Test selection is a decision procedure rather than a test. The question type (comparison with a norm, relationship, prediction, two or more independent or related groups, interactions) is crossed with the design and the outcome's level of measurement. For continuous outcomes meeting parametric assumptions: independent-samples t for two between groups, one-way ANOVA for three or more, factorial ANOVA for several factors; paired t and repeated-measures ANOVA for within designs; split-plot ANOVA for mixed designs. When assumptions fail (independent errors, equal variances, normal errors, interval scale) the alternatives are Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis and Friedman. Nominal outcomes go to chi-square, McNemar for paired proportions, or logistic regression for prediction. The rigour lies in fixing the design and analysis before data collection, using one-tailed tests only for directional hypotheses, and remembering that nonparametric tests lose power and still carry assumptions.

Attributes

Loss type
Active
Stage
Evaluation
Effort
Hours

Steps

  1. State the question type: difference from a norm, relationship, prediction, difference between independent groups, between related measures, or interaction.
  2. Identify the design (between-group, within-group or split-plot), the number of independent variables and the number of conditions for each.
  3. Identify the outcome's data type: continuous, ordinal or nominal.
  4. For continuous outcomes, check independence, homogeneity of variance and normality of residuals.
  5. Read off the parametric test or its nonparametric or categorical alternative from the decision chart, and pre-register the choice where possible.
  6. Lay out the data as the software requires and report the statistic with degrees of freedom, p value and an effect size with confidence interval.

Sources

  • Research Methods in HCI (Lazar et al., 2017)
  • Statistical Analysis in JASP (Goss-Sampson, 2020)
  • Quantitative Psychological Research (Clark-Carter, 2004)
  • Advanced Quantitative Data Analysis (Cramer, 2003)
  • Data Types (Interaction Design Foundation, n.d.)
  • Survey Analysis (Sauro & Lewis, 2021)
  • Confidence Intervals (Sauro & Lewis, 2021)
  • Quantitative Social Research Methods (Singh, 2007)
  • Benchmarking the User Experience (Sauro, 2018)
  • Human-Computer Interaction (Dix et al., 2004)
  • Statistics (Witte & Witte, 2017)
  • Intermediate Statistics for Dummies (Rumsey, 2007)

Also known as

  • Significance test selection by experimental design
  • Test selection by question and data type
  • Table 4.3 test selection
  • Choosing a test for comparing means
  • Which test should I use
  • Statistical test selection

Categories

  • Analytics & Behaviour
  • Statistics & Modelling