Description
Measures, from 0 to 1, how far several judges agree when each ranks the same set of objects, and tests whether their consensus ordering can be trusted.
Kendall's W summarises agreement among m judges who each rank the same n objects. It runs from 0 (no agreement) to 1 (identical rankings) and cannot be negative. A tie-corrected version handles judges who give two objects the same rank and equals the uncorrected value when there are no ties. Its classic use is as a gate: only if independent judges agree significantly can their mean ranks serve as a scale or consensus ordering, for example ranking photographs for attractiveness rather than relying on the researcher's own judgement. Significance comes from exact probabilities, tables for 3-7 objects, or a chi-square approximation for more than 7. Unlike kappa, Krippendorff's alpha or the ICC, it addresses ordinal ranking by several raters directly. A significant W shows agreement beyond chance, not that agreement is high enough for every purpose.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation, Definition
- Data
- Quantitative
- Effort
- Hours
Steps
- Have each of m judges independently rank the same n objects (designs, usability issues, features), allowing ties where unavoidable.
- Compute W, using the tie-corrected formula if any judge has tied ranks.
- Test significance with the exact probability, with tables when n is 3-7, or with the chi-square approximation when n is above 7.
- If agreement is significant, use the mean ranks as the consensus ordering and report W alongside it.
- If agreement is weak, treat the ranking as unreliable: examine which judges or objects diverge and clarify criteria before re-ranking.
Sources
- Quantitative Psychological Research (Clark-Carter, 2004)
Also known as
- Kendall's W
- Coefficient of concordance
- Kendall's coefficient of concordance (W) for agreement among several judges
Categories
- Analytics & Behaviour
- Statistics & Modelling