Anchoring Vignettes

Description

Respondents rate short fixed scenarios on the same scale as themselves, allowing correction for groups using the response scale differently.

When users in different countries or segments rate 'ease of use' on a 1-5 scale, differences may reflect different interpretations of the scale (differential item functioning), not experiences. Anchoring vignettes describe hypothetical people or situations at fixed levels of the construct. Because the vignettes are identical for everyone, differences in how they are rated reveal each group's response scale. Nonparametric methods recode self-ratings relative to vignette ratings; parametric models (CHOPIT) estimate thresholds. This is the leading solution for cross-cultural and cross-segment comparability of self-report scales.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Effort
Weeks

Steps

  1. Write 3-5 vignettes describing fixed levels of the construct (e.g. users with varying difficulty completing a task).
  2. Ask respondents to rate themselves and each vignette on the same scale, randomising vignette order.
  3. Check vignette equivalence and response consistency (vignettes rated in expected order).
  4. Recode self-ratings relative to vignettes (nonparametric C-scores) or fit a CHOPIT model.
  5. Compare groups on corrected scores and report how conclusions change.
  6. Retain vignettes for future waves to monitor scale-use drift.

Sources

  • King, Murray, Salomon & Tandon, Enhancing the validity and cross-cultural comparability of measurement in survey research (2004), American Political Science Review
  • King & Wand, Comparing incomparable survey responses (2007), Political Analysis

Also known as

  • Anchoring vignettes
  • King-Murray vignettes
  • DIF correction with vignettes

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics