Description
Coding recorded pretest interviews for interviewer and respondent behaviours, then flagging questions whose problem-code rate exceeds about 15 per cent for revision.
A quantitative pretest for interviewer-administered questionnaires. Pretest interviews are recorded and every administration of every question is coded with a standard set: interviewer codes (exact reading, minor change, major change, wrong skip) and respondent codes (interruption, request for clarification, inadequate answer, don't know, refusal, adequate answer). Problem-code rates are computed per question, and items above roughly 15 per cent, or high relative to the rest, are flagged once each has at least about ten administrations. Its strength is objectivity and scale: it shows where questions break in real interaction without relying on respondents' introspection. Its limit is that it shows where problems occur, not why, so flagged items are interpreted with coders and interviewers and usually paired with cognitive interviews.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Siloed
- Stage
- Design
- Data
- Mixed
- Effort
- Weeks
Steps
- Record 50 to 100 pretest interviews, with consent.
- Code each question administration for interviewer behaviour (exact reading, minor change, major change, wrong skip).
- Code respondent behaviour (interruption, request for clarification, inadequate answer, don't know, refusal, adequate answer).
- Compute problem-code rates per question.
- Flag questions above about 15 per cent problem codes, or clearly high relative to others, with at least about ten administrations each.
- Debrief coders and interviewers to interpret flagged items, then revise and, where needed, probe them in cognitive interviews.
Sources
- The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
Also known as
- Behavior coding
- Interaction coding
- Fowler behaviour codes
Categories
- Analytics & Behaviour
- Survey & Psychometrics