Mode Effects Analysis in Survey Administration

Description

Checks and designs for how administering a survey by web, app, phone, in person or SMS changes who responds and how they answer.

Mode effects are differences in survey results caused by how a survey is administered. Modes differ in who they reach (selection effects) and in how respondents answer (measurement effects): interviewer-administered modes increase social desirability and acquiescence, aural modes produce recency effects while visual modes produce primacy effects, and small screens increase satisficing and item nonresponse. Mode effects analysis plans for these differences through unified mode design (constructing questions that function equivalently across modes), randomised split-mode experiments to estimate measurement effects, and statistical adjustment when modes are mixed. In UX it is essential when moving surveys between channels, combining in-app and email samples, or reaching low-literacy or offline users by phone, so that trend breaks are not misinterpreted.

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. When a UX survey must be administered in multiple modes (e.g., mobile and desktop): test for mode effects before reporting aggregate results.
  2. Run the same survey on a random split of participants across modes.
  3. For each item, test: Mann-Whitney U test for significant differences between modes.
  4. If significant: report mode-specific results, not aggregates.
  5. Key rule: never compare results collected in different modes without testing for mode effects.

Sources

  • Survey Methodology — Groves et al.
  • Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)

Also known as

  • Mode effects
  • Mixed-mode survey design

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics