Description
Estimates UX metrics from small samples as posterior distributions, using informative or weakly informative priors to give stable, interpretable probability statements.
Bayesian estimation treats unknown quantities, such as a completion rate or the difference in task time between designs, as distributions updated by data. With small samples, frequentist tests are underpowered and produce binary significant or not-significant verdicts; Bayesian models instead yield full posterior distributions, from which teams can read the probability that a metric exceeds a threshold or that one design outperforms another. Weakly informative or evidence-based priors stabilise estimates, and hierarchical models borrow strength across tasks or participants. Modern tools such as PyMC, Stan and brms make this practical. Rigour comes from justifying priors, running prior predictive and sensitivity checks, verifying MCMC convergence, and reporting posteriors rather than only point estimates.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Days
Steps
- Small-sample UX study protocol:
- N < 10 per group → do not use any frequentist significance tests. Report: effect size, bootstrapped 95% CI (10,000 resamples), and Bayes Factor (how much evidence for the effect vs. the null).
- (2) N 10–20 → use permutation tests for significance; Bayesian credible intervals for estimation.
- (3) N > 20 → standard methods apply with distributional checks. R packages: bayestestR (Bayesian estimation), coin (permutation tests), boot (bootstrap).
Sources
- Small Sample Size Solutions — (various)
- PyMC
- scipy.stats
- pingouin
- mord
- bambi
- statsmodels
- matplotlib
- seaborn
- The Inquiry Series, C2 — Quantitative UX Analysis in Python — Descriptive to Inferential
Also known as
- Bayesian data analysis
- Bayesian inference for usability
- Bayesian estimation with PyMC
Categories
- Analytics & Behaviour
- Statistics & Modelling