Kendall's Coefficient of Concordance (W)

Description

Measures, from 0 to 1, how far several judges agree when each ranks the same set of objects, and tests whether their consensus ordering can be trusted.

Kendall's W summarises agreement among m judges who each rank the same n objects. It runs from 0 (no agreement) to 1 (identical rankings) and cannot be negative. A tie-corrected version handles judges who give two objects the same rank and equals the uncorrected value when there are no ties. Its classic use is as a gate: only if independent judges agree significantly can their mean ranks serve as a scale or consensus ordering, for example ranking photographs for attractiveness rather than relying on the researcher's own judgement. Significance comes from exact probabilities, tables for 3-7 objects, or a chi-square approximation for more than 7. Unlike kappa, Krippendorff's alpha or the ICC, it addresses ordinal ranking by several raters directly. A significant W shows agreement beyond chance, not that agreement is high enough for every purpose.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Have each of m judges independently rank the same n objects (designs, usability issues, features), allowing ties where unavoidable.
  2. Compute W, using the tie-corrected formula if any judge has tied ranks.
  3. Test significance with the exact probability, with tables when n is 3-7, or with the chi-square approximation when n is above 7.
  4. If agreement is significant, use the mean ranks as the consensus ordering and report W alongside it.
  5. If agreement is weak, treat the ranking as unreliable: examine which judges or objects diverge and clarify criteria before re-ranking.

Sources

  • Quantitative Psychological Research (Clark-Carter, 2004)

Also known as

  • Kendall's W
  • Coefficient of concordance
  • Kendall's coefficient of concordance (W) for agreement among several judges

Categories

  • Analytics & Behaviour
  • Statistics & Modelling