Description
Estimates the total number of problems (including undetected ones) from the overlap between problems found by independent evaluators or sessions.
Ecologists estimate population size by capturing and marking animals, then recapturing: if few recaptured animals are marked, the population is large. The same logic applies to usability problems found by independent evaluators: if two evaluators find many of the same problems, few remain undiscovered. The Lincoln-Petersen estimator N = n1 x n2 / m, and more robust models (Chao's estimators, models allowing heterogeneous detectability), estimate total problems and hence the proportion found. Software engineering adopted this for code inspections. It provides an evidence-based answer to 'have we found enough?', better than rules of thumb about five users.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Have two or more evaluators or independent session sets identify problems separately.
- Consolidate problems carefully into a master list, with clear matching rules agreed beforehand.
- Build a capture history matrix of which evaluator found which problem.
- Estimate total problems using Lincoln-Petersen (two captures) or Chao and heterogeneity models (multiple captures).
- Compute the proportion discovered and its confidence interval.
- Decide whether to continue evaluation based on estimated undiscovered problems, especially severe ones.
Sources
- Chao, Estimating the population size for capture-recapture data with unequal catchability (1987), Biometrics
- Eick et al., Estimating software fault content before coding (1992), ICSE
- Seber, The Estimation of Animal Abundance and Related Parameters (2nd ed., 1982)
- Quantifying the User Experience (Sauro & Lewis)
Also known as
- Capture-recapture
- Mark-recapture
- Lincoln-Petersen estimator
- Chao estimator
Categories
- Analytics & Behaviour
- Statistics & Modelling