Cochran's Q Test for Related Dichotomous Outcomes

Description

Tests whether the proportion of successes or yes answers differs across three or more conditions when the same participants respond in every condition.

Cochran's Q is the within-subjects test for a binary outcome when there are more than two conditions: the nominal counterpart of one-way repeated-measures ANOVA and the extension of McNemar's test beyond two conditions. Each participant contributes a 0 or 1 in every condition, so typical UX uses are task pass/fail on three or more designs, or whether each participant noticed an element in several layouts. Multi-option categorical answers can be recoded into a set of 0/1 dummies, which also allows a person to pick more than one option. Q is computed from the condition (column) and participant (row) totals, or equivalently as the treatment sum of squares from a within-subjects ANOVA on the 0/1 data divided by the within-subjects mean square. With about 16 or more participants it follows chi-square with k - 1 degrees of freedom; exact probabilities are preferable where software offers them. A significant Q is followed by pairwise McNemar tests with an adjusted alpha.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Code each participant's outcome in every condition as 1 (success or yes) or 0, recoding multi-category answers into 0/1 dummies if needed.
  2. Make sure every participant has a value in every condition; aim for at least 16 participants for the chi-square approximation.
  3. Compute Q from the column and row totals, or as SS(treatments) divided by MS(within subjects) from a repeated-measures ANOVA on the 0/1 data.
  4. Refer Q to chi-square with k - 1 degrees of freedom, or report the exact probability when available.
  5. If Q is significant, compare pairs of conditions with McNemar tests using a Bonferroni-adjusted alpha.
  6. Report the proportion of successes in each condition alongside Q, df and p.

Sources

  • Quantitative Psychological Research (Clark-Carter, 2004)

Also known as

  • Cochran's Q
  • Cochran Q test

Categories

  • Analytics & Behaviour
  • Statistics & Modelling