Description
Present the same decision information in several equivalent formats to random groups; a preference that holds across formats is firm, while a reversal shows the format is deciding.
Rational choice assumes description invariance: equivalent descriptions should yield the same choice. They often do not. McNeil and colleagues gave patients, radiologists and students identical lung-cancer outcomes framed as survival or mortality; radiation was chosen by 42% versus 25% across frames, and physicians were no less affected. Other equivalent formats also flip preferences: complementary percentages (10% versus 5% unemployment against 90% versus 95% employment), transparent versus coalesced layouts (a coalesced layout led 58% to a dominated option), and individual versus group perspective. The check systematically lists the equivalent formats for a decision screen, randomises participants to them and compares choice shares. Robust preferences can be reported as user preferences; format-dependent ones are design problems, since the interface, not the user, is determining the choice.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Days
Steps
- List the equivalent formats for the decision: gain or loss frame, complementary percentages, aggregated or segregated layout, individual or group perspective, named or unnamed options.
- Randomly assign participants to formats, counterbalancing order if several formats are given within person.
- Measure choice shares for each format.
- Test differences and flag decisions whose majority choice flips between formats.
- Report preferences only where they are stable, and treat flips as a design problem (for example showing both frames).
Sources
- Preference, Belief, and Similarity (Tversky, 2004)
- Decision Making: Descriptive, Normative, and Prescriptive Interactions (Bell et al., 1988)
- On Second Thought: Outsmarting Your Mind's Hard-Wired Habits (Herbert, 2010)
- Handbook of Cognition (Lamberts & Goldstone, 2005)
Also known as
- Framing robustness check
- Multiple-format preference check
- Transparent versus opaque format test
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference