Survey data cleaning

Description

Systematically screening raw survey responses for careless, fraudulent, duplicate and out-of-range data using pre-stated rules before any analysis.

Survey cleaning is the first and most consequential analytic step: careless responses typically make up 10-15% of online samples and can create spurious factors, inflate or deflate correlations and distort means. Rigorous cleaning uses multiple indicators: attention or instructed-response items, response time (speeders), long-string analysis (straight-lining), even-odd consistency, Mahalanobis distance for multivariate outliers, duplicate and bot detection, and open-text checks. Rules and thresholds are written before looking at results, applied in code rather than by hand, and every exclusion is counted in a flow table. In UX practice this is often reduced to deleting obviously blank rows, which is why it is vestigial.

Attributes

Loss type
Vestigial
Effort
Days

Steps

  1. Write exclusion rules and thresholds in the analysis plan before inspecting outcomes.
  2. Import the raw export programmatically (e.g. pyreadstat for .sav files) and keep it unchanged.
  3. Flag speeders, commonly those completing in under one third or under half of the median duration.
  4. Flag failed attention checks, long strings exceeding about half the items in a scale block, and multivariate outliers at Mahalanobis distance p below 0.001.
  5. Detect duplicates and bots via IP, device fingerprint, identical open text and impossible response patterns.
  6. Recode out-of-range values to missing, reverse-score items as specified, and document every recode.
  7. Exclude respondents meeting pre-stated combinations of flags and report counts at each step in a flow table; run a sensitivity analysis with and without exclusions.

Sources

  • factor_analyzer
  • semopy
  • pyreadstat (SPSS import)
  • scikit-learn (MICE)
  • pingouin (reliability)
  • pandas
  • Meade, A. W. & Craig, S. B. (2012) Identifying careless responses in survey data. Psychological Methods
  • Curran, P. G. (2016) Methods for the detection of carelessly invalid responses in survey data. JESP
  • The Inquiry Series, C3 — Survey Pipeline in Python — From Raw Responses to Structural Equation Model
  • Benchmarking the User Experience (Sauro)
  • Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)

Also known as

  • Data screening
  • Careless responding detection
  • Response quality screening

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics