Description
A staged process for building a multi-item measure of a UX construct that is reliable, valid and fit for comparison, from item generation to confirmatory testing.
Scale construction turns a construct such as trust in an AI assistant into a defensible measurement instrument. The process defines the construct, generates items from qualitative research and literature, tests content validity with experts, pilots items with cognitive interviews, runs EFA on one sample to refine the structure, confirms it with CFA on another, and then establishes reliability, convergent, discriminant and criterion validity, plus measurement invariance across groups. Established UX instruments such as SUS, UMUX and SUPR-Q followed parts of this path. Rigour lies in each stage being explicit and reported; most homegrown UX surveys skip all of it.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Vestigial
- Stage
- Definition, Evaluation
- Data
- Quantitative
- Effort
- Months
Steps
- Define the construct and its boundaries, distinguishing it from neighbouring constructs.
- Generate an item pool of roughly twice the intended final length from interviews and literature.
- Assess content validity with 5-10 experts (e.g. content validity ratio) and run cognitive interviews with 5-15 target users.
- Administer to a development sample (200 or more) and refine items with EFA and item analysis.
- Confirm the structure with CFA on an independent sample and test invariance across key groups.
- Establish reliability (omega or alpha above 0.7, ideally 0.8) and validity against related measures and behavioural criteria.
- Publish a scoring guide with norms or benchmarks for interpretation.
Sources
- Discovering Statistics Using R/SPSS (Andy Field)
- Quantitative User Experience Research (Chris Chapman & Kerry Rodden)
- Movie Booking Playbook (Vivek Gopinath)
- Scale Development: Theory and Applications (Robert DeVellis)
- Cronbach & Meehl (1955) Construct validity in psychological tests
- Discovering Statistics (Field)
- Quantitative UX Research (Chapman)
- Movie Booking Playbook (Gopinath)
- The Inquiry Series, Ch 50 — Factor Analysis — Discovering the Latent Structure of Experience
Also known as
- Questionnaire development
- Psychometric scale development
- Instrument validation
Categories
- Analytics & Behaviour
- Survey & Psychometrics