Within- and Between-Person Reliability of Momentary Scales

Description

Estimate separately how reliably a multi-item momentary scale detects change within persons and differences between persons, and check its factor structure at each level.

Repeated-measures scales need two reliabilities: one for between-person differences and one for within-person change. A generalisability-theory approach decomposes variance into time, person, item and their interactions, giving RC, the reliability of change; in Bolger and Laurenceau's example a four-item positive affect scale over 10 days had RC = .77. Nezlek's approach fits a three-level model (items in occasions in persons). CFA approaches compute omega at each level from a multilevel CFA or two-step within and between decompositions; the example's within-person omega was .79. Per-person CFAs can also give a distribution of reliabilities. Factor structure can differ by level (for example two factors within persons and one between), so unidimensionality and convergent and discriminant correlations are checked at each level. There is no gold standard, methods differ in handling autocorrelation, and single-item measures cannot be assessed this way.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Include at least three items per key construct in the momentary questionnaire.
  2. Fit a multilevel CFA, or two-step within and between CFAs, or a G-theory time x person x item model.
  3. Compute omega (or RC for change and the corresponding between-person coefficient) at the within and between levels.
  4. Check unidimensionality at each level separately.
  5. Check convergent and discriminant correlations at each level.
  6. Report both reliabilities with the method used, and do not reuse a between-person reliability as evidence of sensitivity to change.

Sources

  • The Open Handbook of Experience Sampling Methodology (Myin-Germeys & Kuppens, 2022)
  • Intensive Longitudinal Methods (Bolger & Laurenceau, 2013)

Also known as

  • Multilevel reliability
  • Multilevel omega
  • Within-person reliability
  • Reliability of within-person change: G-theory and multilevel CFA omega
  • RC coefficient
  • Within-person omega
  • Cranford et al. reliability of change

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics