BERTopic Topic Modelling

Description

Discovering semantically coherent topics in thousands of text segments using sentence embeddings, UMAP, HDBSCAN and class-based TF-IDF, then interpreting them qualitatively.

BERTopic improves on bag-of-words topic models such as LDA by clustering meaning rather than word co-occurrence. Text is chunked into semantic units, embedded with a sentence-transformer model, reduced in dimension with UMAP and clustered with HDBSCAN, which also identifies outliers. Class-based TF-IDF then extracts representative terms per cluster. Parameters are tuned until topics are interpretable, and each topic is named after reading representative excerpts rather than from its keyword list. In qualitative research it serves as a map of large corpora, showing what is talked about and where human attention should go, not as a replacement for interpretive coding.

Attributes

Loss type
Active
Data
Mixed
Effort
Days

Steps

  1. Prepare corpus: Chunk transcripts into semantic units (50–200 words). Remove filler words, researcher turns, timestamps.
  2. Generate sentence embeddings: sentence-transformers (all-MiniLM-L6-v2 sufficient for most UX research corpora).
  3. Run BERTopic: UMAP for dimensionality reduction, HDBSCAN for clustering, c-TF-IDF for topic word extraction.
  4. Evaluate coherence: Review top 10 words per topic. Adjust UMAP/HDBSCAN parameters until topics are interpretable.
  5. Interpret qualitatively: For each topic, retrieve 5–10 representative excerpts. Name the topic based on meaning, not keyword list.
  6. Use as sampling guide: Topics identify where to focus human qualitative analysis — not to replace it.

Sources

  • BERTopic (Maarten Grootendorst)
  • sentence-transformers
  • hdbscan
  • BERTopic: Neural Topic Modeling with a Class-based TF-IDF Procedure (Grootendorst)

Also known as

  • Transformer-based topic modelling
  • Embedding clustering for qualitative data
  • Neural topic modelling

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning