MaxDiff (Best-Worst Scaling)

Description

Respondents repeatedly choose the most and least important items from small subsets, producing ratio-quality importance scores without scale-use bias.

Rating scales suffer from everyone rating everything as important, and from cultural differences in scale use. MaxDiff shows respondents subsets of items (typically 4-5 per screen) from a list of 10-30, asking for the best and worst. Balanced incomplete block designs ensure each item appears equally and with each other item. Hierarchical Bayes multinomial logit estimates individual-level utilities, which can be rescaled to probability shares. The method discriminates strongly and works across cultures. Vestigial use runs MaxDiff in survey tools with poor designs and only counts-based scores, and uses it for things that are not comparable, such as features with very different scopes.

Attributes

Loss type
Vestigial
Effort
Weeks

Steps

  1. Define a list of comparable items (features, needs, messages), worded at similar levels of specificity.
  2. Generate a balanced design: each respondent sees each item about 3 times, 4-5 items per set.
  3. Field to an adequate sample (e.g. 200+ per segment of interest).
  4. Estimate utilities with hierarchical Bayes MNL; check fit (RLH) and remove speeders.
  5. Rescale to probability shares and compare segments with credible intervals.
  6. Optionally add an anchored question to identify absolute importance thresholds.

Sources

  • Louviere, Flynn & Marley, Best-Worst Scaling: Theory, Methods and Applications (2015)
  • Finn & Louviere, Determining the appropriate response to evidence of public concern (1992), Journal of Public Policy & Marketing
  • Factor and Discriminant Analysis (Sauro & Lewis, MeasuringU training)
  • Feature Prioritization Methods (slide deck)

Also known as

  • MaxDiff
  • Best-worst scaling
  • BWS
  • Maximum difference scaling

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics