Species Accumulation and Good-Turing Coverage for Saturation

Description

Estimates how complete a set of discovered themes, codes or problems is from the frequency of items seen only once, giving a quantitative basis for saturation.

Good and Turing, while breaking Enigma codes, showed that the probability of encountering a new type next is estimated by the proportion of observations that are singletons (types seen once). Ecologists use accumulation and rarefaction curves to estimate species richness and sample coverage. Applied to qualitative research, codes or themes are the 'species' and interviews are samples. Coverage estimates (1 minus singletons/total) and extrapolated accumulation curves (e.g. with the iNEXT method) show the expected number of new codes per extra interview. This gives an auditable complement to the vague notion of saturation. Lewis (2001) applied Good-Turing adjustments to usability problem discovery rates.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Code data incrementally and record which codes appear in each interview or session.
  2. Plot the accumulation of unique codes against sample size, randomising order to get a smooth curve.
  3. Count singletons and doubletons; compute Good-Turing coverage and Chao1 richness estimates.
  4. Extrapolate with rarefaction/extrapolation methods (e.g. iNEXT) to predict new codes from additional sessions.
  5. Set stopping criteria (e.g. coverage above 0.9 for themes of interest) before fieldwork.
  6. Report coverage alongside qualitative judgements of meaning saturation.

Sources

  • Good, The population frequencies of species and the estimation of population parameters (1953), Biometrika
  • Chao & Jost, Coverage-based rarefaction and extrapolation (2012), Ecology
  • Lewis, Evaluation of procedures for adjusting problem-discovery rates estimated from small samples (2001), International Journal of Human-Computer Interaction

Also known as

  • Rarefaction curves
  • Good-Turing estimation
  • Sample coverage
  • Accumulation curves

Categories

  • Analytics & Behaviour
  • Statistics & Modelling