ctp-embedder-test

Semantic Message Clustering & Visualization

This project demonstrates how to convert message text into semantic embeddings, cluster them into meaningful groups, and visualize them using UMAP and HDBSCAN.
The result is a clean, interactive, visualization of your dataset.

I plan to use something like this for my cluster data dashboard project.

---

Features

• Embeds text using Hugging Face Sentence Transformers
• Projects embeddings to 2D using UMAP
• Finds natural clusters with HDBSCAN
• Automatically labels clusters using TF-IDF keyword extraction
• Draws cluster boundaries using convex hulls
• Interactive Plotly visualization with hover data
• Loads message data from input.json

---

What I did

1. Embedding Text

We use the Hugging Face embedding model: sentence-transformers/all-MiniLM-L6-v2

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode(texts)

2. UMAP

Embeddings are high-dimensional (usually 384 dimensions), so we reduce them to 2D for visualization.

3. HDBSCAN

HDBSCAN finds clusters automatically based on density, without needing to pick a fixed number of clusters.

4. Automatic Cluster Labels

We use TF-IDF keyword extraction to summarize each cluster’s dominant topics.

5. Interactive Visualization

We render everything using Plotly:

• Points colored by cluster
• Optional “noise” cluster
• Convex hull shading around cluster regions
• Hover tooltips showing text, user, and channel
• Clean layout with hidden axes This creates a highly readable “semantic map” of your dataset.