Quasi-Simplex Test-Retest Reliability for Panel Data

Description

Estimating the reliability of a single survey item from three or more panel waves while allowing the underlying true score to change between waves.

Two-wave test-retest correlations confuse unreliability with real change. The quasi-simplex model separates them with three or more waves: each observed answer is a true score plus random error, and each true score depends on the previous wave's true score (a lag-1 Markov process). With three waves the model is identified by assuming equal reliabilities or equal error variances across waves; with four or more, fit can be tested. Waves must be far enough apart that earlier answers are not remembered. The result is a reliability estimate for a single item, which cross-sectional methods such as Cronbach's alpha cannot give. Assumptions matter: change must follow the lag-1 pattern, errors must be uncorrelated, and memory effects inflate estimates.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Ask the same item, worded identically, in three or more waves spaced far enough apart to prevent recall of earlier answers.
  2. Specify a model in which each wave's answer equals a true score plus random error, and each true score is a function of the previous one.
  3. Identify the model by constraining reliabilities or error variances to be equal across waves.
  4. Fit it in structural equation modelling software; with four or more waves, test model fit.
  5. Report the item's reliability and compare items or formats.

Sources

  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
  • Latent Variable Models (Loehlin & Beaujean, 2017)

Also known as

  • Quasi-Markov simplex model
  • Heise reliability model
  • Multi-wave reinterview reliability

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics