Posterior credible intervals

Description

Reports the range within which a parameter lies with a stated probability given the data and prior, providing intuitive uncertainty statements for UX metrics.

A 95% credible interval contains the parameter with 95% probability given the model and data, which is how most stakeholders already misinterpret frequentist confidence intervals. Credible intervals can be equal-tailed (2.5th to 97.5th percentiles) or highest density intervals, the narrowest range containing 95% of the posterior. Kruschke recommends combining HDIs with a region of practical equivalence (ROPE) to decide whether an effect is meaningfully different from zero. Credible intervals work for any posterior quantity: rates, differences, ratios and predictions. Rigour comes from reporting the prior, checking model fit, using enough posterior samples for stable intervals, and stating the decision rule that links intervals to actions.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Fit a Bayesian model appropriate to the metric, such as Beta-Binomial for rates, log-normal for times or ordinal regression for ratings, with a stated prior.
  2. Draw sufficient posterior samples (for example at least 4,000 effective samples) and check convergence diagnostics such as R-hat below 1.01 where MCMC is used.
  3. Compute the 95% (or 89%, following McElreath) equal-tailed or highest density interval for each quantity of interest.
  4. Define a region of practical equivalence based on the smallest meaningful effect, for example ±1 percentage point in conversion.
  5. Apply the decision rule: effect credible if the HDI lies outside the ROPE, practically equivalent if inside, undecided otherwise.
  6. Report intervals with plain-language statements, such as 'there is a 95% probability the completion rate is between 72% and 89%'.

Sources

  • Statistical Rethinking (McElreath)
  • Bayesian Methods for Interaction & Design
  • Behavioral Drift Playbook (Gopinath)
  • The Inquiry Series, Ch 40 — Bayesian Thinking & Sequential Study Design

Also known as

  • Bayesian credible intervals
  • Highest density intervals
  • Bayesian uncertainty intervals

Categories

  • Analytics & Behaviour
  • Statistics & Modelling