Presentation-invariance (framing robustness) check

Description

Present the same decision information in several equivalent formats to random groups; a preference that holds across formats is firm, while a reversal shows the format is deciding.

Rational choice assumes description invariance: equivalent descriptions should yield the same choice. They often do not. McNeil and colleagues gave patients, radiologists and students identical lung-cancer outcomes framed as survival or mortality; radiation was chosen by 42% versus 25% across frames, and physicians were no less affected. Other equivalent formats also flip preferences: complementary percentages (10% versus 5% unemployment against 90% versus 95% employment), transparent versus coalesced layouts (a coalesced layout led 58% to a dominated option), and individual versus group perspective. The check systematically lists the equivalent formats for a decision screen, randomises participants to them and compares choice shares. Robust preferences can be reported as user preferences; format-dependent ones are design problems, since the interface, not the user, is determining the choice.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. List the equivalent formats for the decision: gain or loss frame, complementary percentages, aggregated or segregated layout, individual or group perspective, named or unnamed options.
  2. Randomly assign participants to formats, counterbalancing order if several formats are given within person.
  3. Measure choice shares for each format.
  4. Test differences and flag decisions whose majority choice flips between formats.
  5. Report preferences only where they are stable, and treat flips as a design problem (for example showing both frames).

Sources

  • Preference, Belief, and Similarity (Tversky, 2004)
  • Decision Making: Descriptive, Normative, and Prescriptive Interactions (Bell et al., 1988)
  • On Second Thought: Outsmarting Your Mind's Hard-Wired Habits (Herbert, 2010)
  • Handbook of Cognition (Lamberts & Goldstone, 2005)

Also known as

  • Framing robustness check
  • Multiple-format preference check
  • Transparent versus opaque format test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference