Description
Using topic coherence metrics to compare topic models and choose the number of topics, combined with human judgement of interpretability.
Topic coherence measures how often a topic's top words co-occur in the corpus or a reference corpus, approximating whether humans would find the topic meaningful. Common measures include UMass, NPMI and C_v (which Röder et al. found correlated best with human ratings). Practitioners fit models across a range of k and plot coherence to find a peak or plateau. Rigour comes from treating coherence as one input: perplexity correlates poorly with interpretability (Chang et al., 2009), coherence can reward generic topics, and it should be paired with stability across seeds, topic diversity and analyst reading of representative documents.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Goal
- Synthesise data
- Stage
- Discovery, Post-launch
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Define the candidate range of topic numbers (e.g. k from 5 to 50 in steps of 5, then finer around promising values).
- Fit models for each k with at least 3 random seeds.
- Compute C_v and NPMI coherence (gensim CoherenceModel) and topic diversity (proportion of unique top words).
- Plot mean coherence with variability against k and identify peaks or plateaus.
- Have two analysts rate interpretability of topics for the 2-3 best candidates, including a word-intrusion test if feasible.
- Choose k based on coherence, stability, diversity and human ratings, and document the decision.
Sources
- gensim (LDA)
- BERTopic
- sentence-transformers
- pyLDAvis (visualisation)
- hdbscan
- Röder, M., Both, A. & Hinneburg, A. (2015) Exploring the space of topic coherence measures. WSDM
- Chang, J. et al. (2009) Reading tea leaves: How humans interpret topic models. NeurIPS
- Mimno, D. et al. (2011) Optimizing semantic coherence in topic models. EMNLP
- The Inquiry Series, C11 — Topic Modelling at Scale — LDA, BERTopic & Semantic Clustering
Also known as
- Topic coherence evaluation
- Choosing number of topics by coherence
Categories
- Analytics & Behaviour
- Computational & Machine Learning