Survey Item-Type Selection (Decision Tree and Rating-Format Catalogue)

Description

Choose each survey item's format by naming its goal first, then its construct, using a decision tree over about fifteen rating, ranking and allocation formats.

Sauro and Lewis classify survey questions as open-ended, closed-ended static (single or multiple choice, rating scales), closed-ended dynamic (adaptive conjoint, MaxDiff, IRT-driven) and task-based. Their catalogue of static rating formats covers linear numeric, Likert agreement, frequency, unipolar and bipolar satisfaction, performance, disconfirmation, comparative, comparative intensity, semantic differential and semantic distance, where a single adjective per row is rated from not at all to perfectly so no polar opposites are needed. Ranking and allocation formats include forced ranking, pick-some (top task) and fixed sum. A decision tree maps goals (discovery, categorical, ranking, allocation, rating, comparison) and constructs to formats, each with documented trade-offs. Format rules follow the evidence: more than three points, a neutral point, positive wording, small grids, counterbalancing. Most format variations shift means by only about 0.3-1.5%, whereas collapsing NPS to three points changed results by 22%.

Attributes

Loss type
Active
Effort
Hours

Steps

  1. State the goal of each item: discovery, categorical measurement, ranking, allocation, rating or comparison.
  2. Discovery: use an open-ended item. Categorical: single choice if options are mutually exclusive, multiple choice if not.
  3. Ranking: forced rank for up to about ten items; pick-some or MaxDiff for more, randomising order. Allocation: fixed sum.
  4. Rating: choose by construct: linear numeric (general), Likert-type (agreement), frequency, unipolar (none to much), bipolar (opposites), performance (adequacy or quality), disconfirmation or Kano (meeting expectations).
  5. Comparison: comparative item for two products; comparative intensity against a named benchmark. Adjectives: semantic differential only if true opposites exist, otherwise semantic distance.
  6. Apply format rules: 5-7 or 11 points, include a neutral point, word positively, keep grids small, counterbalance options and items; record the rationale per item.

Sources

  • Survey Design (Sauro & Lewis, 2021)
  • Survey and Correlational Research Designs (Privitera, 2014)
  • Survey Design (Avedian, 2014)
  • Data-Driven Design: Surveys. Tips for Writing Good Questions (Interaction Design Foundation, n.d.)

Also known as

  • Survey item decision tree
  • 15 common rating scales
  • Four classes of survey questions
  • Semantic distance scale
  • Comparative intensity scale
  • Disconfirmation scale
  • Survey item-type selection decision tree and rating-scale format catalogue

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics