Non-Response Bias Assessment

Description

Checking whether the people who answered a survey differ from those who did not, before generalising the results.

Almost no survey gets answers from everyone it samples. People lack time, motivation or the right knowledge, fill it in wrongly or cannot be reached, so the achieved sample can differ from the population in ways that skew results. Gratton and Jones describe three checks. First, profile the achieved sample on age, gender and other relevant variables, often graphically, to spot unusual patterns. Second, where population figures exist, compare the respondents against them; if one group is under-represented the sample is biased in that respect. Third, send a follow-up (letter or email with another copy of the questionnaire and a fresh cut-off date, worded as a reminder rather than a repeat of the original appeal, and without coercion) and compare those who respond late with the original respondents, treating late responders as a proxy for non-responders. In UX, product data makes this unusually powerful: the user database records plan, tenure, activity level and platform for every sampled person, so respondents can be compared with non-respondents directly, and survey results can be weighted or caveated when the heaviest users are over-represented.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Before fielding, record the characteristics of everyone in the issued sample (from the sampling frame or product data).
  2. After the main cut-off, compare respondents with the issued sample or population on those characteristics, using tables or charts.
  3. Send a follow-up reminder to non-respondents with a new copy or link and a fresh deadline, framed as a reminder and without pressure.
  4. Compare late (follow-up) respondents with early respondents on key answers and characteristics.
  5. Decide whether differences matter for the conclusions; weight the data, restrict claims to the groups that responded, or caveat the findings.
  6. Report response rate and the results of the bias checks alongside the survey findings.

Sources

  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)

Also known as

  • Non-response bias check
  • Respondent versus population comparison
  • Late-respondent comparison
  • Follow-up of non-respondents

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics