Description
Bayesian clustering in which the number of clusters is inferred from the data through a Dirichlet process prior instead of being fixed in advance.
Finite mixtures and latent class models make the analyst fix the number of groups and then compare fits. A Dirichlet process prior instead lets units (users, teams, markets) draw their effects from a discrete distribution with a potentially unlimited number of support points, built by stick-breaking: successive Beta(1, alpha) fractions split the probability among candidate cluster values drawn from a baseline density. The precision alpha governs how readily new clusters form; small values favour few clusters. MCMC then samples cluster allocations and values together, so the output is a posterior distribution over the number of occupied clusters, unit estimates shrunk towards their cluster, and the probability that any two units share a cluster. It is a robust alternative to a single normal random-effects distribution when effects may be lumpy or multimodal.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Goal
- Segment users, Synthesise data
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Choose a baseline density for the unit effects (for example normal on log rates) and decide whether to fix alpha (for example 1 or 5) or give it a gamma prior.
- Set a truncation level comfortably above the number of clusters you expect, for example 10 or 20.
- Build the stick-breaking weights from Beta(1, alpha) fractions, with the last weight taking the remainder, and draw candidate cluster values from the baseline density.
- Sample unit allocations and cluster values by MCMC and check convergence.
- Monitor the number of non-empty clusters, and confirm the truncation level is never fully used.
- Report the posterior distribution of the number of clusters, pairwise co-clustering probabilities and unit effects, and repeat with a different alpha and baseline to check sensitivity.
Sources
- Applied Bayesian Modelling (Congdon, 2003)
- Bayesian Statistical Modelling (Congdon, 2006)
Also known as
- Dirichlet process mixture
- Dirichlet process prior clustering
- Stick-breaking prior
- Nonparametric Bayesian clustering
- Infinite mixture model
- DPP
Categories
- Analytics & Behaviour
- Statistics & Modelling