Bayesian A/B Testing with Expected Loss

Description

Computes posterior distributions for each variant and decides by expected loss of choosing wrongly, rather than by significance, aligning tests with business decisions.

In a Bayesian A/B test, priors (ideally informed by past experiments) are updated with data to posterior distributions of each variant's metric, for example Beta posteriors for conversion. Instead of p-values, the decision quantity is the expected loss: the average amount by which the chosen variant would underperform the best one, integrated over the posterior. Teams stop when expected loss falls below a threshold of caring (e.g. 0.1% of conversion), which ties experiments to the cost of error. Vestigial versions report only 'probability to beat control' with flat priors, ignore metric scale and peek without loss thresholds, recreating the problems they were meant to solve.

Attributes

Loss type
Vestigial
Effort
Days

Steps

  1. Specify priors from historical experiment effect distributions, not flat defaults, and document them.
  2. Define the decision metric and the threshold of caring, expressed in business units.
  3. Update posteriors as data arrive (conjugate models or MCMC for complex metrics).
  4. Compute expected loss for each candidate decision and probability of each variant being best.
  5. Stop and decide when the expected loss of the best option falls below the threshold, or when maximum duration is reached.
  6. Report posterior effect distributions, credible intervals, priors used and sensitivity to prior choice.

Sources

  • Raiffa & Schlaifer, Applied Statistical Decision Theory (1961)
  • Kruschke, Bayesian estimation supersedes the t test (2013), Journal of Experimental Psychology: General
  • Gelman et al., Bayesian Data Analysis (3rd ed., 2013)

Also known as

  • Bayesian A/B testing
  • Expected loss decision rule
  • Probability to be best

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference