Survey paradata analysis

Description

Analyse the by-products of answering (time per question, changed answers, back-navigation, break-offs, device and mode) to find confusing questions, mode problems and careless respondents.

Paradata are records of the response process rather than the responses: how long respondents spent on each question and on the whole survey, whether they changed answers or went back, where they broke off, and which device or mode they used. Survey methodologists use them to locate questions that are confusing (long dwell, many changes), surveys that are too long (break-offs), modes or devices that cause problems, and low-effort responding. Pilot timings set expected answer times, so some tools can warn respondents who answer implausibly fast, a guard against satisficing. Paradata are cheap because most online tools already capture them, but they are personal data and should be disclosed. Fast answers are a signal, not proof, of carelessness; combine with straight-lining checks and attention items before excluding anyone.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Days

Steps

  1. Enable capture of timing, navigation, answer changes and device data in the survey tool, and disclose it in the consent text.
  2. Set expected times per question from the pilot (for example flag responses faster than about 2 seconds per item or under one third of the median total).
  3. Inspect break-off points and unusually long dwell times or frequent answer changes per question.
  4. Compare paradata by mode and device (for example mobile versus desktop) to detect mode-specific problems.
  5. Flag speeders and straight-liners for review alongside other quality checks rather than excluding automatically.
  6. Revise problem questions and layouts in the next wave and compare paradata again.

Sources

  • Understanding Your Users (Baxter et al., 2015)
  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
  • Mail and Internet Surveys (Dillman, 2007)

Also known as

  • Paradata
  • Response-process data analysis
  • Survey process metadata

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics