Description
Run side-by-side and one-at-a-time evaluation conditions, because hard-to-evaluate attributes gain weight only in comparison and preferences can reverse between modes.
Whether options are judged together or alone changes what matters. Under the evaluability principle, attributes whose values are hard to judge in isolation (number of dictionary entries) count only in comparison, while easy-to-judge attributes (a torn cover) dominate in separate evaluation; Hsee's used dictionaries reversed in willingness to pay between modes. Fox and Tversky's comparative ignorance work found that ambiguity aversion appears mainly in joint evaluation: pedestrians pricing bets on San Francisco and Istanbul temperatures showed a gap of about $16 jointly and $1.50 separately. The check runs a joint (within-subject, order counterbalanced) and a separate (between-subject) condition, tests the condition by option interaction, adds a same-source control, and reports the mode that matches how users meet the options in the product.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Unmigrated
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Days
Steps
- Decide whether the real product decision is made side by side or one option at a time.
- If unsure, run a joint condition (each person evaluates both, order counterbalanced) and a separate condition (one option per group).
- Collect valuations, willingness to pay or preferences in both conditions.
- Test the condition by option interaction; flag reversals as evaluability or comparative-ignorance effects.
- Add a control pair from the same source to check the interaction is specific to the hard-to-evaluate or vague option.
- Report results from the design that matches how users actually encounter the options.
Sources
- Heuristics and Biases (Gilovich et al., 2002)
- Preference, Belief, and Similarity (Tversky, 2004)
- On Second Thought: Outsmarting Your Mind's Hard-Wired Habits (Herbert, 2010)
Also known as
- Evaluability check
- Monadic versus comparative evaluation
- Comparative ignorance test
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference