Measurement-error modelling

Description

Treats observed scores as noisy measures of true values, using known standard errors or reliabilities, so regression and path estimates are not distorted by measurement error.

Many UX variables are measured with known uncertainty: a success rate from a handful of participants, a rating averaged over few raters, a scale score with reliability below 1. Plugging such values into a regression as if exact biases coefficients, and not always towards zero. In the Bayesian errors-in-variables form, each true value is a parameter and the observed value is a draw around it with its known standard error, so imprecise points are shrunk and precise points carry more weight. In structural equation modelling, a single scale score becomes the sole indicator of a latent variable with loading 1 and error variance fixed at (1 - reliability) times its variance; this removes error from structural paths without changing fit. Because reliability is uncertain, estimates are rerun over a range of plausible values. Correlated errors must be modelled or the correction overshoots.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Effort
Days

Steps

  1. Draw the causal diagram including the measurement step (true value, error, observed score) and note anything that affects error size, such as sample size per unit.
  2. Obtain error information: standard errors per data point, or a reliability estimate for each scale from the sample, the manual or prior studies.
  3. Bayesian route: declare true values as parameters with observed ~ Normal(true, se), and give error-prone predictors a prior that reflects their own causes.
  4. SEM route: make each single scale score the indicator of a latent variable with loading 1 and error variance fixed to (1 - reliability) x observed variance; give covariates a small error (about 5%) rather than none.
  5. Fit the model, check the sampler or estimation, and compare coefficients with and without the error model; plot observed against estimated true values to see shrinkage.
  6. Run a sensitivity analysis across plausible reliabilities or error sizes and report the source of each reliability value.

Sources

  • Statistical Rethinking (McElreath, 2020)
  • Principles and Practice of SEM (Kline, 2023)
  • Advanced Quantitative Data Analysis (Cramer, 2003)
  • A Beginner's Guide to Structural Equation Modeling (Schumacker & Lomax, 2004)
  • Latent Variable Models (Loehlin & Beaujean, 2017)
  • The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)
  • A User's Guide to Principal Components (Jackson, 1991)
  • Bayesian Statistical Modelling (Congdon, 2006)
  • Nonlinear Regression (Seber & Wild, 1989)
  • Applied Predictive Modeling (Kuhn & Johnson, 2013)
  • Statistical Inference (Casella & Berger, 2001)

Also known as

  • Modelling measurement error in outcomes and predictors
  • Errors-in-variables model
  • Measurement error model
  • Single-indicator measurement-error correction
  • Fixed-reliability single indicator
  • Disattenuation in SEM
  • Single-indicator latent variable

Categories

  • Analytics & Behaviour
  • Statistics & Modelling