Silhouette scores and gap statistic validation

Description

Quantitative checks of whether clusters are compact and well separated, and whether the data has more cluster structure than random noise.

Clustering algorithms always return clusters, even in structureless data, so internal validation is essential. The silhouette compares each point's mean distance to its own cluster with the nearest other cluster, ranging from -1 to 1; averages above 0.5 indicate reasonable structure, above 0.7 strong, and below 0.25 essentially none. The gap statistic compares within-cluster dispersion against that expected under a null reference distribution, choosing the smallest K where the gap is within one standard error of the next. Together with stability checks via bootstrapping, these indices guard against segmenting noise. Rigour requires computing them on the same standardised features used for clustering and reporting them honestly when structure is weak.

Attributes

Loss type
Latent
Stage
Definition
Effort
Days

Steps

  1. Compute average silhouette width for each candidate K and plot silhouette profiles per cluster.
  2. Interpret silhouette: above 0.7 strong, 0.5-0.7 reasonable, 0.25-0.5 weak, below 0.25 no substantial structure.
  3. Compute the gap statistic with at least 100 reference datasets and select the smallest K with Gap(K) at or above Gap(K+1) minus its standard error.
  4. Assess stability by bootstrapping and computing Jaccard similarity per cluster (above 0.75 stable, below 0.5 dissolved).
  5. Report all indices together; if structure is weak, describe a continuum rather than discrete segments.

Sources

  • Cluster Analysis for Researchers (Charles Romesburg)
  • Pattern Recognition and Machine Learning (Christopher Bishop)
  • Methods Atlas (Vivek Gopinath)
  • Rousseeuw (1987) Silhouettes
  • Tibshirani, Walther & Hastie (2001) Estimating the number of clusters via the gap statistic
  • Cluster Analysis for Researchers
  • Pattern Recognition & ML (Bishop)
  • Methods Atlas (Gopinath)
  • The Inquiry Series, Ch 51 — Behavioral Clustering & Probabilistic Segmentation
  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • Internal cluster validation
  • Cluster validity indices
  • Cluster validity assessment
  • Silhouette analysis
  • Segmentation validation
  • Cluster Validation
  • Adjusted Rand index comparison

Categories

  • Analytics & Behaviour
  • Statistics & Modelling