Description
A reliability coefficient for continuous ratings that tells you how consistently targets are scored across judges or items, in agreement or consistency forms.
The intraclass correlation treats judges (or items) as a within-subjects factor and expresses reliability as the share of rating variance due to real differences between targets. Shrout and Fleiss distinguish cases: ICC(2,1) treats judges as a random sample and counts their mean differences against reliability (absolute agreement); ICC(3,1) treats judges as fixed and ignores mean shifts (consistency). Average-measure forms, ICC(2,k) and ICC(3,k), apply when the mean of k judges is the score you will use. The choice changes the answer markedly: in Maxwell et al.'s example the same data give ICC(2,1) = .29 and ICC(3,1) = .71. The same coefficient also appears in mixed models as the proportion of variance due to a random factor such as participant or team. Rigour lies in choosing the form by the question, not by which number is larger.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Siloed
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Arrange the data as targets (rows) by judges or items (columns), with every judge rating every target where possible.
- Decide whether systematic differences in judge leniency should count against reliability (agreement) or not (consistency).
- Choose Case 2 (random judges, agreement) if judges stand for a wider pool, or Case 3 (fixed judges, consistency) if these judges are the only ones that matter.
- Choose the single-measure form if one judge's score will be used, or the average-measure form if the mean of k judges will be used.
- Compute from the two-way ANOVA mean squares (or a mixed model) and report the form, point estimate and 95% confidence interval.
- Interpret against conventional bands (roughly below .50 poor, .50-.75 moderate, .75-.90 good, above .90 excellent) and improve rater training or anchors if low.
Sources
- Designing Experiments and Analyzing Data (Maxwell et al., 2018)
- The Open Handbook of Experience Sampling Methodology (Myin-Germeys & Kuppens, 2022)
Also known as
- ICC
- Intraclass correlation for rater or item reliability
- Shrout and Fleiss ICC
- ICC(2,1)
- ICC(3,1)
Categories
- Analytics & Behaviour
- Survey & Psychometrics