Description
A structured checklist for deciding whether a statistic, survey result or study finding reported by someone else is trustworthy enough to act on.
Most statistical abuse is not bad arithmetic but subtle exaggeration, so the consumer must ask who produced a figure, what type of statistic it is, how the data were gathered and what was left out. Rumsey's checks cover arithmetic consistency (percentages totalling 100), missing or tiny sample sizes, raw counts used where rates are needed, distorted charts, non-random and self-selected samples, low response rates, leading questions, absent margins of error (which cover sampling error only, not bias), causal claims from observational data, selective reporting of one significant result among many, and anecdotes. Her example of crime totals rising while the rate per 100,000 fell shows how two sides can both cite correct figures. For UX, the checklist applies to vendor benchmarks, industry reports, stakeholder slides and internal dashboards before they drive decisions.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Strategy, Evaluation
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Identify who produced the statistic and whether they have an interest in the result.
- Check the arithmetic: percentages that should total 100 do, counts add to the total and fractions match percentages.
- Find the sample size and number of actual respondents; treat results with no n or a tiny n as unreliable.
- Check that the statistic fits the question, using rates rather than counts when the base changes.
- Examine charts for honest scales, starting points, equal intervals and no 3-D distortion.
- Check how the sample was chosen and the response rate, and read the actual question wording for leading phrasing.
- Look for a margin of error, and if a causal claim is made, check whether it comes from a randomised experiment or an observational study with confounders.
- Ask how many analyses were run and whether data were dropped; wait for independent confirmation of surprising findings.
Sources
- Statistics for Dummies (Rumsey, 2003)
Also known as
- Spotting misleading statistics
- Critiquing reported survey and poll results
- Statistics consumer checklist
- Statistical claim audit
Categories
- Research Operations
- Research Design & Rigor