Reliability analysis (Cronbach's alpha, split-half, omega)

Description

Estimating how consistently the items of a UX scale measure the same construct, to judge whether composite scores are trustworthy enough for decisions.

Internal consistency reliability estimates the proportion of score variance that is true-score variance. Split-half reliability correlates two halves of a scale and corrects with the Spearman-Brown formula; Cronbach's alpha is the average of all possible split-halves under strict assumptions. Alpha assumes tau-equivalence (equal loadings) and unidimensionality; when these fail it underestimates reliability or masks multidimensionality, so McDonald's omega from a factor model is now recommended. Thresholds commonly used are 0.70 for research and 0.90 or above for high-stakes individual decisions. Alpha also rises with item count, so very high values (above 0.95) can indicate redundant items. In UX, alpha is often quoted without checking dimensionality, which makes the practice vestigial.

Attributes

Loss type
Vestigial
Stage
Evaluation
Effort
Hours

Steps

  1. Confirm unidimensionality of the item set (via EFA or CFA) before computing reliability.
  2. Reverse-code negative items and run RELIABILITY in SPSS (/MODEL=ALPHA and /MODEL=SPLIT) or pingouin.cronbach_alpha in Python.
  3. Compute McDonald's omega from a one-factor model where possible and report alongside alpha with confidence intervals.
  4. Review corrected item-total correlations, flagging items below about 0.30, and 'alpha if item deleted' values.
  5. Interpret against purpose: about 0.70 for group comparisons, 0.80 for applied research, 0.90 or higher for individual decisions; treat above 0.95 as possible redundancy.
  6. Report reliability for each sample and segment, since it is a property of scores in a population, not of the instrument.

Sources

  • SPSS FACTOR
  • RELIABILITY procedures; syntax automation; parallel analysis via Python plugin
  • Cronbach, L. J. (1951) Coefficient alpha and the internal structure of tests. Psychometrika
  • McDonald, R. P. (1999) Test Theory: A Unified Treatment
  • Sijtsma, K. (2009) On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika
  • Field, A. (2024) Discovering Statistics Using IBM SPSS Statistics, 6th ed.
  • The Inquiry Series, C7 — SPSS for Survey Analysis — EFA, Reliability & Scale Validation
  • Garcia-Ruiz, M. (ed.), Games User Research: A Case Study Approach (2016)

Also known as

  • Internal consistency reliability
  • RELIABILITY procedure

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics