Description
Ranks units within every posterior draw of their scores, then reports each unit's probability of holding each rank, so claims like 'best' or 'worst' carry honest uncertainty.
League tables of designs, features, teams or products present a single order, but if the underlying scores are uncertain, so is the ordering. Given posterior draws of each unit's score from any Bayesian model (a latent factor, an ideal point, a group mean, an item parameter), the units are sorted within each draw. Storing those ranks over many draws gives a probability distribution over each unit's rank, over which unit holds a given rank, or over complete orderings. The method tested a widely reported claim that one US senator was the most liberal: in the posterior no senator was unambiguously most liberal, the leader topping the list in about 30% of draws and the named senator in about 11%. Similarly, among US states on a social-indicators factor, the most likely worst-off state held that position with probability of only .30. When units are tightly bunched they share rank mass, which is the honest answer rather than a failure.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Stage
- Evaluation, Strategy
- Data
- Quantitative
- Effort
- Hours
Steps
- Fit a Bayesian model that yields posterior draws of the scores to be ranked (for example satisfaction by product area, success rate by design, latent quality by vendor).
- For each saved draw, rank the units' scores.
- Tabulate, for each unit, the share of draws in which it holds each rank, and the probability it is first, last or in the top k.
- Plot rank probability distributions for the units of interest.
- Report probabilities for the specific claim (for example the probability that design C is best) instead of a single ordered list.
- Use enough draws for stable small probabilities and check sensitivity of rank probabilities to the priors.
Sources
- Bayesian Analysis for the Social Sciences (Jackman, 2009)
- Applied Bayesian Modelling (Congdon, 2003)
- Bayesian Statistical Modelling (Congdon, 2006)
Also known as
- Ranking with uncertainty
- Posterior probability of being ranked first
- Bayesian league table
- Inference over rank orderings
- Posterior rank probabilities for latent scores (order-statistic inference)
Categories
- Analytics & Behaviour
- Statistics & Modelling