Strength-versus-weight evidence weighting experiment

Description

Vary evidence strength (how extreme) and weight (sample size, reliability) independently and regress judgements on both, showing whether people neglect how much evidence there is.

Evidence has strength, such as the proportion of heads or the average star rating, and weight, such as the number of tosses or reviews. A normative judge weights both appropriately; people overweight strength. In the coin-bias study, 12 combinations of proportion and sample size (3 to 33 tosses) were judged with an accuracy prize; strength outweighed weight for 30 of 35 respondents, with a median ratio of about 2 to 1. The result is overconfidence when strength is high and weight low, and underconfidence when weight is high and strength low. Regressing the log odds of each judgement on log strength and log weight per person quantifies the imbalance. Within-subjects designs draw attention to the varied cue, so between-subjects arms show more neglect.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Choose the two cues, for example average rating (strength) and number of reviews (weight).
  2. Build a factorial set that decorrelates them.
  3. Ask for a probability, confidence or choice per stimulus, ideally with an accuracy incentive.
  4. Compute the normative answer where one exists.
  5. Regress judgement log odds on log strength and log weight per person and compare coefficients.
  6. Plot calibration by weight level and confirm key effects in between-subjects arms.

Sources

  • Preference, Belief, and Similarity (Tversky, 2004)
  • Griffin, D., & Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive Psychology, 24

Also known as

  • Evidence strength and credence test
  • Griffin-Tversky confidence regression

Categories

  • Analytics & Behaviour
  • Behavioral Science & Economics