Validity/reliability as formal properties

Description

Treats reliability and validity as quantifiable properties of a measure, estimated with specific statistics, rather than labels claimed for a method.

Reliability is the consistency of a measure: internal consistency (Cronbach's alpha or McDonald's omega), test-retest stability (intraclass correlation), and inter-rater agreement (Cohen's kappa, Krippendorff's alpha). Validity is the degree to which evidence supports interpreting scores as measuring the intended construct; modern psychometrics (Messick; the APA/AERA/NCME Standards) treats it as a unified argument drawing on content, response process, internal structure, relations to other variables and consequences. In UX, 'validated' is often claimed for any published questionnaire regardless of the population, and bespoke surveys are used without any checks. The formal approach estimates these properties for each use, sets thresholds in advance, and reports them.

Attributes

Loss type
Vestigial
Stage
Definition
Effort
Weeks

Steps

  1. Specify the intended score interpretation and use, and the population.
  2. Estimate internal consistency (alpha or omega; 0.70 minimum for research, 0.90 for individual decisions) and, if relevant, test-retest reliability over 1-2 weeks (ICC above 0.75).
  3. For coded data, estimate inter-rater agreement on at least 10-20% of items (kappa or Krippendorff's alpha of 0.67-0.80 or higher).
  4. Gather structural validity evidence with factor analysis (sample of at least 150-200; CFA fit such as CFI above 0.95, RMSEA below 0.06).
  5. Gather convergent and discriminant evidence by correlating with related and unrelated measures.
  6. Write a validity argument summarising evidence, gaps and limits on interpretation.

Sources

  • Dix, A., Statistics for HCI: Making Sense of Quantitative Data (2020)
  • Sauro, J. & Lewis, J. R., Quantifying the User Experience (2nd ed., 2016)
  • Leavy, P., Research Design: Quantitative, Qualitative, Mixed Methods, Arts-Based, and Community-Based Participatory Research Approaches (2017)
  • Statistics for HCI (Alan Dix)
  • Quantifying the User Experience (Sauro & Lewis)
  • Research Design (Leavy)
  • The Inquiry Series, Ch 8 — Measurement Theory — Before You Measure Anything

Also known as

  • Psychometric validation
  • Reliability and validity assessment
  • Measurement quality analysis

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics