Description
Running a small-scale version of a survey under real conditions before launch to check wording, order, timing, administration and analysis.
A pilot survey is a small administration of the full questionnaire before the main survey. Gratton and Jones give it five jobs: checking that wording is clear and unambiguous to the intended respondents; checking that the sequence is logical; estimating completion time; testing the whole administration process from distribution to receipt of responses; and doing a dry run of the analysis by entering the pilot data and trying to answer the research question with it. The last is the step most often skipped, and it is where researchers discover that a question cannot support the comparison they need. The pilot should run in conditions as close to the main survey as possible and with a similar group (if the main sample includes children, so must the pilot). After changes, the questionnaire is piloted again until no further changes are needed. It differs from cognitive interviewing, which probes how individuals understand items one by one; a pilot tests the instrument and the fielding pipeline end to end. For UX teams launching in-product or email surveys, a pilot to a small random slice catches broken logic, unclear items and unusable data before the whole user base is surveyed once and cannot be asked again.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Vestigial
- Stage
- Discovery, Evaluation
- Data
- Mixed
- Effort
- Days
Steps
- Prepare the questionnaire, invitation and delivery mechanism exactly as they will be used.
- Field it to a small group similar to the target sample, in conditions as close as possible to the main survey (for example a small random slice of users).
- Record completion times, drop-off points and any questions respondents skip, query or misread; optionally ask pilot respondents to comment on the design.
- Enter or export the pilot data and run the planned analysis to confirm the data can answer the research question.
- Revise wording, order, response options, routing and administration based on what you find.
- Repilot after changes and repeat until no further changes are needed, then launch.
Sources
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
Also known as
- Pilot survey
- Survey pilot
- Questionnaire pretest
- Dry-run survey administration
Categories
- Analytics & Behaviour
- Survey & Psychometrics