Survey Question Writing (Diagnostic Checks, Structures and Wording Principles)

Description

A checklist procedure for turning draft queries into questions every respondent interprets the same way, can answer accurately and is willing to answer.

Question writing treated as a procedure rather than a craft. Each draft item is first put through eight diagnostic checks (does it apply to everyone, do people hold a ready answer, can they recall it, will they reveal it, will categories or layout sway it, is more than one mode used). The writer then chooses a structure deliberately: open-ended for facts, closed ordered for evaluations, closed unordered for comparisons, partially closed with an 'Other' option. Nineteen wording principles follow, such as numbers instead of vague quantifiers, balanced scales, no double-barrelled or yes-means-no items, and no respondent arithmetic. Sauro and Lewis add psycholinguistic checks: avoid embedded clauses and passive voice, and watch words such as 'ever', 'should', an ambiguous 'you' and a sponsoring 'our'. Dillman's demonstration that category ranges alone moved 23% to 69% of students into a top study-time band shows why the rigour matters.

Attributes

Loss type
Vestigial
Effort
Days

Steps

  1. List each draft query with its response choices and the decision it serves.
  2. Ask the eight diagnostic checks of every item: universal applicability, ready-made answer, recall, willingness to reveal, motivation, influence of categories or layout, mode differences, and whether the sponsor will accept changes.
  3. Choose the structure deliberately: open-ended with units printed after the blank for facts, closed ordered for evaluations, closed unordered for comparisons (limit rankings to a top three), or partially closed with 'Other, please specify'; split mixed items into ordered items plus one comparison item.
  4. Apply the wording principles: simple and few words, complete sentences, numbers instead of vague quantifiers, balanced positive and negative categories, undecided placed last, both sides of the scale in the stem, mutually exclusive categories, appropriate time referents, no double-barrelled or yes-means-no items, and no percentages to calculate.
  5. Replace check-all-that-apply lists with a yes/no or rating per option.
  6. Run psycholinguistic checks: remove centre- and left-embedded clauses, passive voice and adjunct wh-questions, and flag watch words (ever, always, never, could, should, would, you, our).
  7. Have colleagues review the set and pilot it with a way for respondents to flag unclear items, then revise.

Sources

  • Mail and Internet Surveys (Dillman, 2007)
  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
  • Survey Design (Sauro & Lewis, 2021)
  • Survey and Correlational Research Designs (Privitera, 2014)
  • Survey Design (Avedian, 2014)
  • Data-Driven Design: Surveys. Tips for Writing Good Questions (Interaction Design Foundation, n.d.)

Also known as

  • Dillman's 19 principles for writing survey questions
  • Question wording principles
  • Survey question writing checklist
  • Interview and survey question wording checks
  • Question wording checks
  • Designing questions and questionnaires

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics