Staged-choice (agenda) effect test

Description

Give random groups the same options with different staged agendas (which options are grouped first) and compare final choice shares to measure how navigation structure steers choice.

When choices are made in stages (category first, then item), the grouping can change the outcome. Tversky and Sattath used triples where two options were similar and one dissimilar. Half the participants first chose between the similar pair and the odd option (an agenda matching the similarity structure); half chose between a pair that split the similar options. The incompatible agenda raised the share of the weaker similar option and lowered the dissimilar one; theory says choice is invariant to the agenda only when the agenda matches users' preference tree, and a worked example moves a nearly dominated option from almost zero to 0.22. In interfaces, filters, category trees and comparison steps are agendas, so this test shows whether information architecture is steering what users end up choosing.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Choose three options with a known similarity structure, checking similarity with user ratings rather than team assumptions.
  2. Create a compatible agenda (the similar pair grouped together first) and an incompatible one (a pair that splits the similar options).
  3. Randomly assign participants and record the final choice after both stages; keep stage labels neutral so they carry no information.
  4. Compare final choice shares between agendas, and compute a within-person index if each person completes several triples (the original used 10 triples per person and about 100 participants).
  5. Redesign category or filter structures whose agenda shifts shares away from what users choose under a compatible agenda.

Sources

  • Preference, Belief, and Similarity (Tversky, 2004)

Also known as

  • Agenda effect test
  • Constrained choice order test
  • Category-first versus item-first test

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference