Description
Organising topics into a tree of broad themes and narrower sub-topics, so large text corpora can be explored at the right level of detail for each decision.
Flat topic models force a single granularity. Hierarchical approaches produce a tree: either generatively (hLDA) or, more commonly today, by agglomerating fine-grained BERTopic clusters using the similarity of their c-TF-IDF or embedding representations. Executives can see a handful of broad themes while product teams drill into specific sub-topics. Rigour comes from choosing a linkage method, validating that merges are semantically sensible by reading documents, keeping the tree shallow enough to interpret (typically 2-3 levels), and labelling each level with consistent conventions. The hierarchy is an analytic aid, analogous to arranging codes into categories in qualitative analysis.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Goal
- Synthesise data
- Stage
- Post-launch, Discovery
- Data
- Qualitative
- Effort
- Days
Steps
- Fit a fine-grained topic model (e.g. BERTopic with a small min_cluster_size giving 50-200 topics).
- Compute the topic hierarchy (BERTopic hierarchical_topics with a chosen linkage, e.g. Ward or average).
- Visualise the dendrogram and choose 2-3 cut levels that give interpretable groupings.
- Read representative documents at each merge to confirm semantic coherence, and override merges that are wrong.
- Label parent and child topics consistently and record definitions.
- Report prevalence at each level and map leaf topics to owning teams or journeys.
Sources
- gensim (LDA)
- BERTopic
- sentence-transformers
- pyLDAvis (visualisation)
- hdbscan
- Blei, D. M., Griffiths, T. L. & Jordan, M. I. (2010) The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. JACM
- Grootendorst, M. (2022) BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794
- The Inquiry Series, C11 — Topic Modelling at Scale — LDA, BERTopic & Semantic Clustering
Also known as
- Topic hierarchies
- Nested topic models
Categories
- Analytics & Behaviour
- Computational & Machine Learning