Description
Rules for sequencing a questionnaire like a conversation, plus a review of adjacent items for documented order effects such as assimilation, contrast and evenhandedness.
A questionnaire is read as a conversation: abrupt topic switches make the sponsor seem not to listen and produce top-of-the-head answers. Dillman orders topics by salience to the respondent, matching the theme promised in the invitation, places objectionable items last, sequences within topics chronologically with descriptive items before evaluative ones, and groups items that share a response format. The first question gets special care: it should apply to everyone, be easy and interesting, and visibly connect to the stated purpose. The rigorous part is a check of adjacent opinion items for five documented effects: norm of evenhandedness, anchoring, carryover (addition), subtraction, and summary items answered after specific ones. A classic split ballot moved agreement with letting communist reporters into the US from 37% to 73% purely by order, and such effects appear in self-administered surveys too.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Latent
- Stage
- Design
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Group questions by topic and order topics from most to least salient, matching the purpose stated in the invitation.
- Place objectionable or sensitive items (income, health, law-breaking, politics) near the end.
- Within each topic, ask in chronological and logical order, with descriptive items before evaluative ones, and group items that share response categories.
- Choose a first question that applies to everyone, is easy and interesting and clearly connected to the purpose; avoid demographics, open-ended items and ranking tasks first. If screening is needed, make it the first question.
- Review every adjacent pair of opinion items for evenhandedness, anchoring, carryover, subtraction and general-after-specific effects.
- Where an effect is suspected, separate the items, put general before specific questions, or randomise order across forms and compare.
- Record order-sensitive pairs so later waves keep the same order.
Sources
- Mail and Internet Surveys (Dillman, 2007)
- Survey Design (Avedian, 2014)
- Data-Driven Design: Surveys. Tips for Writing Good Questions (Interaction Design Foundation, n.d.)
Also known as
- Questionnaire question sequencing
- Order effects check
- Context effects in questionnaires
- First-question selection
Categories
- Analytics & Behaviour
- Survey & Psychometrics