Description
Tests whether the proportion of successes or yes answers differs across three or more conditions when the same participants respond in every condition.
Cochran's Q is the within-subjects test for a binary outcome when there are more than two conditions: the nominal counterpart of one-way repeated-measures ANOVA and the extension of McNemar's test beyond two conditions. Each participant contributes a 0 or 1 in every condition, so typical UX uses are task pass/fail on three or more designs, or whether each participant noticed an element in several layouts. Multi-option categorical answers can be recoded into a set of 0/1 dummies, which also allows a person to pick more than one option. Q is computed from the condition (column) and participant (row) totals, or equivalently as the treatment sum of squares from a within-subjects ANOVA on the 0/1 data divided by the within-subjects mean square. With about 16 or more participants it follows chi-square with k - 1 degrees of freedom; exact probabilities are preferable where software offers them. A significant Q is followed by pairwise McNemar tests with an adjusted alpha.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Code each participant's outcome in every condition as 1 (success or yes) or 0, recoding multi-category answers into 0/1 dummies if needed.
- Make sure every participant has a value in every condition; aim for at least 16 participants for the chi-square approximation.
- Compute Q from the column and row totals, or as SS(treatments) divided by MS(within subjects) from a repeated-measures ANOVA on the 0/1 data.
- Refer Q to chi-square with k - 1 degrees of freedom, or report the exact probability when available.
- If Q is significant, compare pairs of conditions with McNemar tests using a Bonferroni-adjusted alpha.
- Report the proportion of successes in each condition alongside Q, df and p.
Sources
- Quantitative Psychological Research (Clark-Carter, 2004)
Also known as
- Cochran's Q
- Cochran Q test
Categories
- Analytics & Behaviour
- Statistics & Modelling