Description
Systematically checks a survey's items and scales for wording problems, reliability, validity, response patterns and data quality before or after fielding.
Many UX surveys use home-made scales with untested wording, unvalidated item groupings and unknown reliability. A scale quality audit evaluates surveys through several lenses. Item wording is checked for double-barrelled, leading, ambiguous or jargon-laden items and for balanced, labelled response options. Cognitive interviews test whether respondents understand items as intended. Psychometric checks on fielded data examine internal consistency (Cronbach's alpha or, better, McDonald's omega, with about 0.70 as a common minimum), item-total correlations, factor structure, and floor and ceiling effects. Data quality checks look for straightlining, speeding and inattentive responding. Groves' total survey error framework connects these to coverage, sampling, non-response and measurement error. Rigour comes from combining qualitative pre-testing with quantitative psychometric evidence.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Latent
- Stage
- Definition, Evaluation
- Data
- Mixed
- Effort
- Weeks
Steps
- Review each item against a checklist: one idea per item, neutral wording, no jargon, clear time frame, balanced and fully labelled response options, and consistent scale direction.
- Run cognitive interviews with 5 to 15 respondents using think-aloud and verbal probes ('What does "reliable" mean to you here?') and revise problem items.
- After fielding, compute item distributions and flag floor or ceiling effects, for example more than 15% of responses at either extreme.
- Calculate reliability (omega or alpha, targeting 0.70 or above, and investigating values above about 0.95 for redundancy) and corrected item-total correlations (flag below 0.30).
- Check structure with exploratory or confirmatory factor analysis where samples allow (often 200 or more), and test validity against related measures or behaviour.
- Screen data quality: straightlining across grids, completion times below a plausible minimum (for example under one-third of the median), and failed attention checks.
- Replace weak home-made scales with validated instruments where possible (for example SUS, UMUX-Lite, SUPR-Q) and document changes.
Sources
- Statistics for Behavioral Sciences
- Discovering Statistics via SPSS (Field)
- Survey Methodology (Groves)
- The Inquiry Series, Ch 42 — Ordinal Logic — Treating Likert & Survey Data Correctly
Also known as
- Scale validation audit
- Psychometric audit
- Questionnaire quality review
Categories
- Analytics & Behaviour
- Survey & Psychometrics