Cophenetic and Matrix Correlation Checks for Hierarchical Clustering

Description

Correlate pairwise resemblances with tree-implied distances, and compare matrices or opposite linkage methods, to check whether a hierarchical clustering distorts the data or produces artefactual clusters.

A family of checks on whether a dendrogram faithfully represents its data. Tracing each pair of objects to the level where they first join gives the cophenetic matrix; unstringing it and the original resemblance matrix into lists and computing Pearson's r gives the cophenetic correlation, with about 0.8 or more commonly accepted (UPGMA tends to score highest). The same matrix-correlation move compares any two resemblance matrices over the same objects: two measures, two user groups, two coefficients, or a hypothesis-predicted tree against the observed one; many analyses can themselves be clustered to see which agree. A third check clusters the same matrix with single linkage, which chains, and complete linkage, which resists chaining: if the opposite methods give much the same tree the clusters are well defined, and if not they may be artefacts. These checks turn card-sort or similarity dendrograms from pictures into evidence.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Compute the resemblance matrix and cluster it with the chosen method (for example UPGMA).
  2. Derive the cophenetic matrix by recording, for every pair, the level at which they first join in the tree.
  3. Unstring the lower triangles of the resemblance and cophenetic matrices into two lists, plot them, and compute Pearson's r; treat about 0.8 or above as acceptable distortion.
  4. Re-cluster the same matrix with single linkage and complete linkage and compare the trees visually or by correlating their cophenetic matrices; treat groups stable across both as real and investigate groups found in only one.
  5. To compare two measures, groups or coefficients, correlate their resemblance matrices (preferred over cophenetic matrices) against a tolerance set in advance.
  6. Report the coefficients with scatter plots, noting that pairs are not independent so no ordinary significance test applies.

Sources

  • Cluster Analysis for Researchers (Romesburg, 1984)

Also known as

  • Cophenetic correlation coefficient
  • Cophenetic matrix
  • Tree distortion index
  • Matrix correlation between resemblance matrices
  • Matrix correlation coefficient
  • Opposite-linkage agreement check (SLINK versus CLINK)
  • Single versus complete linkage comparison
  • Chaining check

Categories

  • Analytics & Behaviour
  • Statistics & Modelling