Bimodal population identification

Description

Detects whether a metric's distribution conceals two or more distinct user populations, then estimates their sizes and characteristics instead of reporting one misleading average.

An average of a bimodal distribution describes nobody: if novice investors take 3 minutes and experts 40 seconds, a 'typical' 1 minute 50 seconds is a fiction. Bimodal population identification tests for multiple modes and models the data as a mixture of component distributions. Hartigan's dip test checks for unimodality; Gaussian mixture models fitted to log-transformed values, compared via BIC across 1 to 4 components, estimate how many groups exist, their proportions and parameters. Each user's probability of belonging to each component can then be linked to behavioural and demographic attributes to identify who the groups are. Rigour comes from requiring both statistical support and substantive interpretation, checking stability across samples, and avoiding over-fitting components that reflect noise.

Attributes

Loss type
Latent
Stage
Post-launch
Effort
Days

Steps

  1. Plot a histogram and kernel density of the metric (log-transformed if skewed), with several bandwidths to avoid artefacts.
  2. Run Hartigan's dip test; p below 0.05 indicates departure from unimodality, though with very large samples small dips may be significant.
  3. Fit Gaussian mixture models with 1 to 4 components to the (log) values and select the model with the lowest BIC, requiring a BIC improvement of about 10 or more for strong support.
  4. Assign users to components by posterior probability, keeping only confident assignments (for example above 0.8) for profiling.
  5. Profile components by joining with attributes such as tenure, device, region and feature use, and interview 5 to 8 users from each component to interpret them.
  6. Replicate on a different time window or sample, then report components separately with their sizes and characteristics.

Sources

  • Quantifying the User Experience (Sauro & Lewis)
  • Statistics for HCI (Dix)
  • Practical Statistics for Data Scientists (Bruce)
  • The Inquiry Series, Ch 39 — The Distribution of Human Performance — Why Averages Lie
  • van der Maas, H., Complex Systems Research in Psychology (2024)

Also known as

  • Mixture modelling of user metrics
  • Hidden subpopulation detection
  • Multimodality testing

Categories

  • Analytics & Behaviour
  • Statistics & Modelling