Bland-Altman Agreement Analysis

Description

Assesses whether two measurement methods agree well enough to be interchangeable by plotting differences against means and computing limits of agreement.

Correlation is the wrong statistic for comparing measurement methods: two methods can correlate at 0.95 yet differ systematically by a large amount. Bland and Altman proposed plotting the difference between methods against their mean for each subject, computing the mean difference (bias) and the 95% limits of agreement (bias plus or minus 1.96 standard deviations of differences), and judging whether those limits are acceptable for practical use. Proportional bias appears as a slope. In UX this applies whenever a cheaper measure is proposed to replace a gold standard: unmoderated versus moderated task time, webcam versus lab eye tracking, AI-generated ratings versus human ratings, short versus long scales.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Effort
Hours

Steps

  1. Measure the same participants or items with both methods.
  2. Compute differences and means for each pair.
  3. Plot differences against means; calculate bias and 95% limits of agreement with confidence intervals.
  4. Check for proportional bias (regress differences on means) and heteroscedasticity; use log transformation or ratio-based limits if needed.
  5. Compare limits with a pre-specified acceptable range based on practical decision impact.
  6. Conclude interchangeability only if limits fall within the acceptable range.

Sources

  • Bland & Altman, Statistical methods for assessing agreement between two methods of clinical measurement (1986), The Lancet
  • Altman & Bland, Measurement in medicine: the analysis of method comparison studies (1983), The Statistician
  • Tenenbaum, G. & Eklund, R. (eds), Handbook of Sport Psychology, 3rd ed. (2007)

Also known as

  • Bland-Altman plot
  • Limits of agreement
  • Method comparison study

Categories

  • Analytics & Behaviour
  • Statistics & Modelling