Scale construction and validation

Description

A staged process for building a multi-item measure of a UX construct that is reliable, valid and fit for comparison, from item generation to confirmatory testing.

Scale construction turns a construct such as trust in an AI assistant into a defensible measurement instrument. The process defines the construct, generates items from qualitative research and literature, tests content validity with experts, pilots items with cognitive interviews, runs EFA on one sample to refine the structure, confirms it with CFA on another, and then establishes reliability, convergent, discriminant and criterion validity, plus measurement invariance across groups. Established UX instruments such as SUS, UMUX and SUPR-Q followed parts of this path. Rigour lies in each stage being explicit and reported; most homegrown UX surveys skip all of it.

Attributes

Loss type
Vestigial
Effort
Months

Steps

  1. Define the construct and its boundaries, distinguishing it from neighbouring constructs.
  2. Generate an item pool of roughly twice the intended final length from interviews and literature.
  3. Assess content validity with 5-10 experts (e.g. content validity ratio) and run cognitive interviews with 5-15 target users.
  4. Administer to a development sample (200 or more) and refine items with EFA and item analysis.
  5. Confirm the structure with CFA on an independent sample and test invariance across key groups.
  6. Establish reliability (omega or alpha above 0.7, ideally 0.8) and validity against related measures and behavioural criteria.
  7. Publish a scoring guide with norms or benchmarks for interpretation.

Sources

  • Discovering Statistics Using R/SPSS (Andy Field)
  • Quantitative User Experience Research (Chris Chapman & Kerry Rodden)
  • Movie Booking Playbook (Vivek Gopinath)
  • Scale Development: Theory and Applications (Robert DeVellis)
  • Cronbach & Meehl (1955) Construct validity in psychological tests
  • Discovering Statistics (Field)
  • Quantitative UX Research (Chapman)
  • Movie Booking Playbook (Gopinath)
  • The Inquiry Series, Ch 50 — Factor Analysis — Discovering the Latent Structure of Experience

Also known as

  • Questionnaire development
  • Psychometric scale development
  • Instrument validation

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics