Document Clustering with BERT

Deep language-model representations for unsupervised document clustering.

Searching large corpora of publications is slow and tedious. This project builds document-clustering models that use BERT and Sentence-BERT (SBERT) embeddings to automatically group documents by topic — turning a manual reading task into an instant, unsupervised one.

🔍 Overview

We investigate several variations of a pre-trained BERT model to find which best produces word/document embeddings for representing documents within a larger corpus. The embeddings are:

1. Generated from multiple BERT variants (base BERT, mean-pooled, last-four-layers concatenated, and SBERT)
2. Reduced in dimensionality with PCA
3. Clustered with K-Means to reveal the underlying topics in a corpus

Key finding: across the tested variants, SBERT produced the best document representations for clustering and for capturing document-to-document similarity.

🧪 Approach

Step	Method
Embedding	BERT (base, mean-pooled, last-4-layer concat) and SBERT
Dimensionality reduction	PCA
Clustering	K-Means
Similarity validation	Cosine similarity across manually grouped similar / dissimilar documents

We hypothesized that BERT embeddings could detect similarity between documents from their pre-trained representations, and validated this by measuring cosine similarity within a group of similar files versus a group of dissimilar files. SBERT consistently gave the clearest separation.

📁 Repository structure

NLP_Final_Project_Code.ipynb     # Main: BERT embeddings → PCA → K-Means clustering
BERT Cosine Similarity Test.ipynb # Validates BERT/SBERT similarity on grouped documents
data/                             # Pre-computed embeddings (BERT variants, SBERT) + raw data
Project_report.pdf                # Full write-up and results
requirments.txt                  # Python dependencies

🚀 Getting started

Prerequisites: Python 3.7+ and Jupyter Notebook.

pip install -r requirments.txt

Then open the notebooks:

• NLP_Final_Project_Code.ipynb — evaluates BERT embeddings for clustering. Embeddings are pre-computed and stored as CSVs in ./data, reduced with PCA, and clustered with K-Means.
• BERT Cosine Similarity Test.ipynb — tests how well BERT/SBERT embeddings capture similarity between documents using cosine similarity.

📊 Results

SBERT outperformed the other BERT variants at producing embeddings that cleanly separate topics under PCA + K-Means, and at distinguishing similar from dissimilar documents by cosine similarity. See Project_report.pdf for the full analysis.

---

Built by Rakshit Makan — Generative AI / NLP engineer. Originally developed as an NLP research project (CSCI 6509, Deep Language Model Representation of Document Clustering).