Total Survey Error Framework

Description

Audits a survey for every source of error (coverage, sampling, nonresponse, measurement, processing) rather than sampling error alone, to design and interpret UX surveys defensibly.

The total survey error (TSE) framework, associated with Robert Groves, decomposes the difference between a survey estimate and the true population value into distinct sources. On the representation side are coverage error (the frame misses parts of the population), sampling error, and nonresponse error; on the measurement side are validity problems, measurement error from question wording, mode and respondents, and processing error in coding and editing. Applied to UX surveys, TSE shifts attention from margin of error, which reflects only sampling, to the often larger biases from who is invited and who responds and how questions are understood. Rigour comes from auditing each source at design time, collecting paradata, and testing for bias with follow-up samples.

Attributes

Loss type
Unmigrated
Effort
Weeks

Steps

  1. TSE audit for any UX survey:
  2. Coverage error — who is excluded from the sampling frame? Users who never complete tasks? Low-literacy users?
  3. (2) Sampling error — is the sample representative of the population? Compute design effect if cluster sampling.
  4. (3) Nonresponse error — who declines to respond? Is there a pattern?
  5. (4) Measurement error — do items measure what they intend? (cognitive interview pre-testing).
  6. (5) Processing error — coding/data entry errors. Estimate the magnitude of each error type and prioritize the largest.

Sources

  • Survey Methodology — Groves et al.

Also known as

  • TSE
  • Survey error decomposition

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics