Intraclass correlation (ICC)

Description

A reliability coefficient for continuous ratings that tells you how consistently targets are scored across judges or items, in agreement or consistency forms.

The intraclass correlation treats judges (or items) as a within-subjects factor and expresses reliability as the share of rating variance due to real differences between targets. Shrout and Fleiss distinguish cases: ICC(2,1) treats judges as a random sample and counts their mean differences against reliability (absolute agreement); ICC(3,1) treats judges as fixed and ignores mean shifts (consistency). Average-measure forms, ICC(2,k) and ICC(3,k), apply when the mean of k judges is the score you will use. The choice changes the answer markedly: in Maxwell et al.'s example the same data give ICC(2,1) = .29 and ICC(3,1) = .71. The same coefficient also appears in mixed models as the proportion of variance due to a random factor such as participant or team. Rigour lies in choosing the form by the question, not by which number is larger.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Arrange the data as targets (rows) by judges or items (columns), with every judge rating every target where possible.
  2. Decide whether systematic differences in judge leniency should count against reliability (agreement) or not (consistency).
  3. Choose Case 2 (random judges, agreement) if judges stand for a wider pool, or Case 3 (fixed judges, consistency) if these judges are the only ones that matter.
  4. Choose the single-measure form if one judge's score will be used, or the average-measure form if the mean of k judges will be used.
  5. Compute from the two-way ANOVA mean squares (or a mixed model) and report the form, point estimate and 95% confidence interval.
  6. Interpret against conventional bands (roughly below .50 poor, .50-.75 moderate, .75-.90 good, above .90 excellent) and improve rater training or anchors if low.

Sources

  • Designing Experiments and Analyzing Data (Maxwell et al., 2018)
  • The Open Handbook of Experience Sampling Methodology (Myin-Germeys & Kuppens, 2022)

Also known as

  • ICC
  • Intraclass correlation for rater or item reliability
  • Shrout and Fleiss ICC
  • ICC(2,1)
  • ICC(3,1)

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics