Cognitive Interviewing for Questionnaire Design

Description

Think-aloud and verbal probing with a few respondents to reveal how they comprehend, recall, judge and map answers to survey items before fielding.

Cognitive interviewing tests survey questions against how people actually answer them. Following Tourangeau's model, answering involves comprehension, retrieval from memory, judgement or estimation, and mapping the answer onto the response format; each stage can fail. In a cognitive interview, a small sample of target respondents answers draft items while thinking aloud or responding to scripted and spontaneous probes such as 'what does this question ask?' or 'how did you arrive at that?'. Problems are coded by stage and items failing for several participants are revised and retested. It is routine in national statistics offices and catches ambiguous terms, vague quantifiers, inaccessible memories and ill-fitting scales that no statistical analysis can fix after fielding.

Attributes

Loss type
Siloed
Stage
Design
Effort
Days

Steps

  1. Recruit 5–10 participants from your target population. Think-aloud or verbal probing — choose one consistently.
  2. For each question: ask the participant to answer it while thinking aloud, then probe on comprehension, retrieval strategy, and response mapping.
  3. Code problems by type. Revise items that show comprehension, retrieval, or response problems in 3+ participants.
  4. Re-test revised items with a new subsample before final deployment.

Sources

  • Analysis of the Cognitive Interview in Questionnaire Design
  • Survey Methodology (Groves et al.)
  • International Handbook of Survey Methodology
  • Statistics for Behavioral Sciences
  • Discovering Statistics via SPSS (Field)
  • Survey Methodology (Groves)
  • Cognitive Interviewing: A Tool for Improving Questionnaire Design (Willis)
  • The Psychology of Survey Response (Tourangeau, Rips & Rasinski)
  • The Inquiry Series, Ch 42 — Ordinal Logic — Treating Likert & Survey Data Correctly

Also known as

  • Cognitive pretesting
  • Cognitive interview
  • Verbal probing for surveys

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics