Everyday Statistical-Reasoning Problem Set

Description

Give short everyday scenarios where sample size, variability or regression matters, code open answers as deterministic, poor or good statistical, and compare conditions.

Fong, Krantz and Nisbett built eighteen open-ended problems crossing three problem types (with an obvious random device, objective without randomness cues, and subjective judgements) with six structures, including generalising from a small sample, regression to the mean and a large but biased sample used to catch over-application. Answers are coded 1 for deterministic, 2 for poor statistical and 3 for good statistical reasoning using a coding guide; four coders agreed exactly on 86% of answers. The set measures whether training, examples or presentation formats increase statistical reasoning, and a disguised telephone 'sports survey' showed transfer outside class. For UX, the same structure can test whether dashboards, experiment readouts or explanations lead stakeholders or users to account for sample size and regression.

Attributes

Loss type
Unmigrated
Data
Mixed
Effort
Weeks

Steps

  1. Write scenario problems for each combination of problem type and structure, including false-alarm items with large but biased samples.
  2. Randomise order so no two problems with the same structure are adjacent.
  3. Collect written or recorded explanations rather than multiple-choice answers.
  4. Code each answer 1, 2 or 3 with a coding guide, blind to condition, and check inter-coder agreement.
  5. Code false alarms where sample-size criticism is wrongly applied.
  6. Compare conditions on the frequency and quality of statistical answers; for transfer, embed items in an unrelated survey later.

Sources

  • Decision Making: Descriptive, Normative, and Prescriptive Interactions (Bell et al., 1988)

Also known as

  • Law-of-large-numbers reasoning test
  • Statistical heuristics test (Fong, Krantz and Nisbett)
  • Disguised-survey transfer test of statistical reasoning
  • Everyday statistical-reasoning problem set with open-response coding

Categories

  • User Research
  • Cognitive Psychology & Perception