Description
Discovers latent topics in large text corpora such as reviews, tickets and survey comments without predefined categories, enabling qualitative synthesis at scale.
LDA models each document as a mixture of topics and each topic as a probability distribution over words, fitted to explain word co-occurrence. The treatise's transfer: apply to app reviews, support transcripts, open-ended survey responses and NPS comments, where thousands to millions of items exceed manual coding. LDA discovers latent problem themes without pre-specifying categories; outputs need interpretive labelling but the structure is data-driven. Structural topic models add covariates (rating, platform, date) so topic prevalence can be compared across segments and time. Rigour requires preprocessing decisions, choosing K via coherence and held-out likelihood, and validating topics by reading representative documents. Embedding-based topic models are modern alternatives.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Latent
- Goal
- Synthesise data
- Stage
- Post-launch, Discovery
- Data
- Mixed
- Effort
- Days
Steps
- Assemble and clean the corpus (deduplicate, remove boilerplate, handle languages).
- Tokenise, remove stop words and build a vocabulary.
- Fit models across a range of K and assess coherence and exclusivity.
- Label topics by reading top words and the most representative documents.
- Model topic prevalence against covariates (rating, version, segment) with STM if needed.
- Validate a sample manually and feed topics into deeper qualitative follow-up.
Sources
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Blei DM, Ng AY & Jordan MI (2003) Latent Dirichlet allocation. JMLR 3:993-1022
- Roberts ME, Stewart BM & Tingley D (2014) Structural topic models for open-ended survey responses. AJPS
- gensim (LDA)
- BERTopic
- sentence-transformers
- pyLDAvis (visualisation)
- hdbscan
- Blei, D. M., Ng, A. Y. & Jordan, M. I. (2003) Latent Dirichlet Allocation. Journal of Machine Learning Research
- Röder, M., Both, A. & Hinneburg, A. (2015) Exploring the space of topic coherence measures. WSDM
- The Inquiry Series, C11 — Topic Modelling at Scale — LDA, BERTopic & Semantic Clustering
- A Practical Guide to Sentiment Analysis (Erik Cambria et al.)
- Artificial Intelligence for Humans (Jeff Heaton)
- Pattern Recognition and Machine Learning (Christopher Bishop)
- Blei, Ng & Jordan (2003) Latent Dirichlet Allocation
- Practical Guide to Sentiment Analysis
- AI for Humans (Heaton)
- Pattern Recognition (Bishop)
- The Inquiry Series, Ch 55 — Natural Language at Scale — Computational Qualitative Analysis
Also known as
- Topic modelling
- LDA
- Structural topic model (STM)
- LDA topic modelling
- Probabilistic topic models
- Latent Dirichlet Allocation for large corpora
- LDA topic model
- Latent Dirichlet Allocation (LDA for topics)
Categories
- Analytics & Behaviour
- Computational & Machine Learning