Bradley-Terry Paired Comparison Scaling

Description

Estimates a latent strength for each item from many pairwise choices, handling incomplete designs, ties and covariates with uncertainty.

The Bradley-Terry model sets the probability that item i beats item j as pi_i / (pi_i + pi_j), equivalent to a logistic model on strength differences. It accommodates incomplete comparison designs (not every pair compared), allows home-advantage or position effects, and extends to ties, covariates and random effects. It underpins comparative judgement in educational assessment and human preference modelling for language models (RLHF reward models). For UX, it lets you scale many design variants, AI outputs or content items from simple pairwise preferences collected adaptively, with confidence intervals rather than raw win rates.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Define items and the comparison question (e.g. 'Which response is more helpful?').
  2. Design comparisons: random or adaptive pairing so the comparison graph is connected; randomise left/right.
  3. Collect judgements from raters, recording rater and position.
  4. Fit a Bradley-Terry model (BradleyTerry2, choix) with position effects and optional rater random effects.
  5. Report strengths with confidence intervals and check model fit and intransitivities.
  6. Use adaptive pairing to refine estimates among close competitors.

Sources

  • Bradley & Terry, Rank analysis of incomplete block designs: I. The method of paired comparisons (1952), Biometrika
  • Agresti, Categorical Data Analysis (3rd ed., 2013)
  • Turner & Firth, Bradley-Terry models in R: the BradleyTerry2 package (2012), Journal of Statistical Software
  • Feature Prioritization Methods (slide deck)

Also known as

  • Bradley-Terry model
  • Paired comparison model
  • Elo-style scaling
  • Comparative judgement

Categories

  • Analytics & Behaviour
  • Statistics & Modelling