Hierarchical topic modelling

Description

Organising topics into a tree of broad themes and narrower sub-topics, so large text corpora can be explored at the right level of detail for each decision.

Flat topic models force a single granularity. Hierarchical approaches produce a tree: either generatively (hLDA) or, more commonly today, by agglomerating fine-grained BERTopic clusters using the similarity of their c-TF-IDF or embedding representations. Executives can see a handful of broad themes while product teams drill into specific sub-topics. Rigour comes from choosing a linkage method, validating that merges are semantically sensible by reading documents, keeping the tree shallow enough to interpret (typically 2-3 levels), and labelling each level with consistent conventions. The hierarchy is an analytic aid, analogous to arranging codes into categories in qualitative analysis.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Fit a fine-grained topic model (e.g. BERTopic with a small min_cluster_size giving 50-200 topics).
  2. Compute the topic hierarchy (BERTopic hierarchical_topics with a chosen linkage, e.g. Ward or average).
  3. Visualise the dendrogram and choose 2-3 cut levels that give interpretable groupings.
  4. Read representative documents at each merge to confirm semantic coherence, and override merges that are wrong.
  5. Label parent and child topics consistently and record definitions.
  6. Report prevalence at each level and map leaf topics to owning teams or journeys.

Sources

  • gensim (LDA)
  • BERTopic
  • sentence-transformers
  • pyLDAvis (visualisation)
  • hdbscan
  • Blei, D. M., Griffiths, T. L. & Jordan, M. I. (2010) The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. JACM
  • Grootendorst, M. (2022) BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794
  • The Inquiry Series, C11 — Topic Modelling at Scale — LDA, BERTopic & Semantic Clustering

Also known as

  • Topic hierarchies
  • Nested topic models

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning