Integrated Classification Likelihood (ICL) and Entropy Criteria for the Number of Clusters

Description

Choose the number of segments by penalising the mixture likelihood for complexity and for how fuzzy the resulting classification is, favouring well-separated clusters.

BIC chooses the number of mixture components that best describes the density, which can mean several overlapping components fitting one skewed group. When the goal is clustering, the entropy of the posterior membership probabilities matters: it is near zero when each case clearly belongs to one component and large when components overlap. ICL-BIC adds twice this entropy to BIC, so it prefers solutions with confidently classified, well-separated segments. Related criteria are CLC (minus twice the log likelihood plus twice the entropy, which overestimates the count when proportions are unequal), NEC (entropy divided by the log-likelihood gain over one component, accepting more than one cluster only if it falls below 1) and AWE. In McLachlan and Ng's simulations ICL and ICL-BIC recovered the true number of groups where BIC did not.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Fit the mixture for each candidate number of components by EM from several starts, keeping the best admissible solution.
  2. Compute each case's posterior membership probabilities and the entropy EN = minus the sum of tau log tau over cases and components.
  3. Compute ICL-BIC = -2 log L + 2 EN + d log n, where d is the number of free parameters, and optionally NEC.
  4. Choose the number of components that minimises ICL-BIC; with NEC accept more than one cluster only if NEC is below 1.
  5. Report ICL alongside BIC and treat disagreement as a sign of overlapping components.

Sources

  • Finite Mixture Models (McLachlan & Peel, 2000)

Also known as

  • ICL
  • ICL-BIC
  • Classification likelihood criterion (CLC)
  • Normalised entropy criterion (NEC)
  • Approximate weight of evidence (AWE)
  • Entropy-penalised BIC

Categories

  • Analytics & Behaviour
  • Statistics & Modelling