Interleaved experiments

Description

Merge two rankers' result lists into one alternating list shown to each user and credit clicks to the source algorithm, giving a highly sensitive paired comparison.

Comparing ranking algorithms with ordinary A/B tests needs large samples because between-user variance swamps small differences. Interleaving shows each user a single list built from both rankers: items alternate between the two lists, duplicates are removed, and which algorithm picks first is balanced (team-draft interleaving). Each click is credited to the algorithm that supplied the item, and preferences are aggregated across queries. Because every user sees both algorithms, the comparison is paired and can be one to two orders of magnitude more sensitive than an A/B test, making it a fast screening step before a full A/B test on business metrics. It is valid only when result slots are homogeneous and the change affects nothing outside the list.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Generate both ranked lists for each query or request.
  2. Interleave them alternately, removing duplicates and randomising or balancing which algorithm contributes first (team-draft).
  3. Log which algorithm supplied each displayed item.
  4. Credit clicks (or other engagement) to the supplying algorithm per query.
  5. Aggregate wins across queries and test whether the preference differs from 50%.
  6. Confirm promising rankers with a standard A/B test on the overall evaluation criterion.

Sources

  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • Designing Machine Learning Systems (Huyen, 2022)

Also known as

  • Interleaved experiment for ranking changes
  • Interleaving
  • Team-draft interleaving

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference