Description
Estimates the uncertainty of any UX metric by resampling the observed data with replacement thousands of times, giving confidence intervals without strong distributional assumptions.
The bootstrap treats the sample as a stand-in for the population: drawing many resamples of the same size with replacement and recomputing the statistic gives its sampling distribution. Intervals can then be formed for medians of task time, differences in SUS scores, ratios of conversion rates or model coefficients, all of which lack simple formulas or violate normality. The BCa (bias-corrected and accelerated) interval generally performs best. For UX, the bootstrap makes honest uncertainty reporting accessible even for small usability samples, though with very small n (under about 10-15) intervals are unreliable. Rigour requires resampling at the right unit (participants, not individual tasks, when tasks are nested) and enough resamples.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Latent
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Identify the statistic of interest (e.g. median task time or the difference in completion rates).
- Identify the independent resampling unit, usually the participant or user, and resample at that level.
- Draw at least 2,000 resamples (10,000 for percentile tails or BCa) with replacement.
- Recompute the statistic for each resample and build the BCa or percentile 95% interval.
- Report the point estimate with the interval and note sample size limitations, especially below around 15 participants.
Sources
- Monte Carlo Methods with R (Christian Robert & George Casella)
- Diffusion of Innovations (Everett Rogers)
- Forward-Thinking Playbook (Vivek Gopinath)
- Efron (1979) Bootstrap methods: another look at the jackknife
- An Introduction to the Bootstrap (Bradley Efron & Robert Tibshirani)
- scipy.stats and pingouin (Python)
- Monte Carlo Methods with R (Robert & Casella)
- Diffusion of Innovations (Rogers)
- Forward-Thinking Playbook (Gopinath)
- The Inquiry Series, Ch 56 — Monte Carlo Simulation & Predictive Forecasting
- scipy.stats
- pingouin
- mord
- PyMC
- bambi
- statsmodels
- matplotlib
- seaborn
- The Inquiry Series, C2 — Quantitative UX Analysis in Python — Descriptive to Inferential
- Practical Statistics for Data Scientists (Bruce & Bruce)
Also known as
- Bootstrap confidence intervals
- Bootstrapping
- Resampling inference
- Jackknife
- Residual bootstrap
- Paired (case) bootstrap
Categories
- Analytics & Behaviour
- Statistics & Modelling