Description
Reports the range within which a parameter lies with a stated probability given the data and prior, providing intuitive uncertainty statements for UX metrics.
A 95% credible interval contains the parameter with 95% probability given the model and data, which is how most stakeholders already misinterpret frequentist confidence intervals. Credible intervals can be equal-tailed (2.5th to 97.5th percentiles) or highest density intervals, the narrowest range containing 95% of the posterior. Kruschke recommends combining HDIs with a region of practical equivalence (ROPE) to decide whether an effect is meaningfully different from zero. Credible intervals work for any posterior quantity: rates, differences, ratios and predictions. Rigour comes from reporting the prior, checking model fit, using enough posterior samples for stable intervals, and stating the decision rule that links intervals to actions.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Latent
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Fit a Bayesian model appropriate to the metric, such as Beta-Binomial for rates, log-normal for times or ordinal regression for ratings, with a stated prior.
- Draw sufficient posterior samples (for example at least 4,000 effective samples) and check convergence diagnostics such as R-hat below 1.01 where MCMC is used.
- Compute the 95% (or 89%, following McElreath) equal-tailed or highest density interval for each quantity of interest.
- Define a region of practical equivalence based on the smallest meaningful effect, for example ±1 percentage point in conversion.
- Apply the decision rule: effect credible if the HDI lies outside the ROPE, practically equivalent if inside, undecided otherwise.
- Report intervals with plain-language statements, such as 'there is a 95% probability the completion rate is between 72% and 89%'.
Sources
- Statistical Rethinking (McElreath)
- Bayesian Methods for Interaction & Design
- Behavioral Drift Playbook (Gopinath)
- The Inquiry Series, Ch 40 — Bayesian Thinking & Sequential Study Design
Also known as
- Bayesian credible intervals
- Highest density intervals
- Bayesian uncertainty intervals
Categories
- Analytics & Behaviour
- Statistics & Modelling