Beta-Binomial Problem Discovery Model

Description

Models variation in how often different problems occur, so estimates of how many problems remain are not overly optimistic.

The classic 1 − (1 − p)^n model assumes every problem has the same chance of being found. In reality some problems hit almost everyone and some very few, and this heterogeneity (overdispersion) makes the simple model overestimate how much has been found: you think you are done when you are not. The beta-binomial model lets p itself vary according to a beta distribution, with a mean and a spread parameter. Schmettow's simulations found it fitted real discovery curves better in three of five datasets. His advice, reported by Sauro and Lewis: use Lewis's adjusted binomial for small studies (n under 6), switch to the beta-binomial once you have 10 or more participants, and keep a generous safety margin when usability is mission-critical. A zero-truncated logit-normal binomial is a related alternative.

Attributes

Loss type
Unmigrated
Stage
Evaluation
Data
Mixed
Effort
Hours

Steps

  1. Record a participant-by-problem matrix from a formative study of at least about 10 participants.
  2. Fit a beta-binomial (or zero-truncated logit-normal binomial) to the problem frequencies.
  3. Estimate the mean and variability of p.
  4. Project the discovery curve and the expected number of unseen problems.
  5. Compare with the simple adjusted binomial; if they differ, plan more sessions.

Sources

  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • Expanded binomial model
  • Logit-normal binomial problem discovery
  • Overdispersion-aware discovery model

Categories

  • User Research
  • Usability & Evaluation