Description
Systematically screening raw survey responses for careless, fraudulent, duplicate and out-of-range data using pre-stated rules before any analysis.
Survey cleaning is the first and most consequential analytic step: careless responses typically make up 10-15% of online samples and can create spurious factors, inflate or deflate correlations and distort means. Rigorous cleaning uses multiple indicators: attention or instructed-response items, response time (speeders), long-string analysis (straight-lining), even-odd consistency, Mahalanobis distance for multivariate outliers, duplicate and bot detection, and open-text checks. Rules and thresholds are written before looking at results, applied in code rather than by hand, and every exclusion is counted in a flow table. In UX practice this is often reduced to deleting obviously blank rows, which is why it is vestigial.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Vestigial
- Goal
- Synthesise data
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Write exclusion rules and thresholds in the analysis plan before inspecting outcomes.
- Import the raw export programmatically (e.g. pyreadstat for .sav files) and keep it unchanged.
- Flag speeders, commonly those completing in under one third or under half of the median duration.
- Flag failed attention checks, long strings exceeding about half the items in a scale block, and multivariate outliers at Mahalanobis distance p below 0.001.
- Detect duplicates and bots via IP, device fingerprint, identical open text and impossible response patterns.
- Recode out-of-range values to missing, reverse-score items as specified, and document every recode.
- Exclude respondents meeting pre-stated combinations of flags and report counts at each step in a flow table; run a sensitivity analysis with and without exclusions.
Sources
- factor_analyzer
- semopy
- pyreadstat (SPSS import)
- scikit-learn (MICE)
- pingouin (reliability)
- pandas
- Meade, A. W. & Craig, S. B. (2012) Identifying careless responses in survey data. Psychological Methods
- Curran, P. G. (2016) Methods for the detection of carelessly invalid responses in survey data. JESP
- The Inquiry Series, C3 — Survey Pipeline in Python — From Raw Responses to Structural Equation Model
- Benchmarking the User Experience (Sauro)
- Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
- Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
Also known as
- Data screening
- Careless responding detection
- Response quality screening
Categories
- Analytics & Behaviour
- Survey & Psychometrics