Web Probing and Respondent Debriefing

Description

Adding a few open probes to a fielded or pilot survey and coding the answers at scale to learn how respondents interpreted specific items.

Field-based probing bridges cognitive interviews and quantitative pretests. Instead of a handful of lab interviews, a few targeted open probes are embedded in a pilot or live survey and answered by hundreds of respondents. Three probe types are used: category-selection probes (why did you choose that answer), comprehension probes (what does this term mean to you) and specific probes (what did you have in mind when answering). Probes can follow the item or come in a debriefing block at the end, and are randomised across respondents so each sees only a few. Open answers are coded into interpretation themes per item, which reveal misreadings, differences between groups or languages, and the frame of reference people used. It scales well online but adds burden and break-off, and written answers are often short.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Days

Steps

  1. Select the items of concern, such as new, translated or ambiguous questions.
  2. Write category-selection, comprehension or specific probes for each and decide whether they follow the item or sit in an end-of-survey debrief.
  3. Randomise probes across respondents so each answers only a few.
  4. Field with the pilot or main sample.
  5. Code open answers into interpretation themes per item and compare across segments or languages.
  6. Revise items, or document interpretation differences that explain group differences.

Sources

  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)

Also known as

  • Web probing
  • Embedded probing
  • Respondent debriefing
  • Category selection probe
  • Field-based probing

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics