distilbert-jd-classifier

distilbert-jd-classifier

Fine-tuned distilbert-base-uncased that classifies a job description into one of four engineering tracks:

• ai_ml
• devops_sre
• hardware_embedded
• fullstack_swe

Built to solve a real problem: given a job posting, automatically pick which resume variant to send.

Results (held-out test set, n=40)

A note on the data quality lesson this project produced

The training set (400 rows, pulled from a Kaggle job-postings dataset plus an HF full-text search over a 33k-row job dataset for FPGA/VLSI/embedded roles) initially trained to 0.75 macro-F1. An audit of the worst misclassifications surfaced 3 mislabeled rows — staffing-agency "Account Manager" / "Recruiter" postings that got swept into hardware_embedded because they mentioned FPGA/embedded as one of many industries the agency covers, even though the job itself has nothing to do with engineering.

Removing those 3 rows (0.75% of the data) took macro-F1 from 0.75 → 0.90. Code for the audit step: scripts/audit_labels.py in the GitHub repo.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="srijavuppala22/distilbert-jd-classifier", top_k=None)
clf(full_job_description_text, truncation=True, max_length=512)
# [{'label': 'ai_ml', 'score': 0.83}, {'label': 'devops_sre', 'score': 0.09}, ...]

Pass the full job description text, not a one-line summary — see Limitations below.

Training details

• Base model: distilbert-base-uncased
• 3 epochs, lr=2e-5, batch size 16, max_length=512
• Selection metric: macro-F1 (not accuracy — classes are imbalanced enough that accuracy is misleading)
• Trained locally on an Apple M3 (MPS backend), ~2 min/run

Limitations

• Training data is 100% public/generic postings (Kaggle + HF), not real job-search data — expect some accuracy loss on postings with unusual phrasing or hybrid roles (e.g. "MLOps Engineer" genuinely straddles ai_ml and devops_sre).
• 4-class only; anything outside these tracks (sales, PM, data analyst, etc.) will be forced into the nearest of the 4.
• Trained on full-length JD text (hundreds of words); confidence drops sharply on short one-line inputs. A one-sentence paraphrase of a job (e.g. "hiring an ML engineer for RAG pipelines") produces near-uniform, low-confidence scores across all 4 classes because it's out of distribution from what the model saw during training. Always pass the complete posting text.
• On the 40-example held-out test set, macro-F1 measured between 0.85 and 0.90 across repeated eval runs (pipeline tokenization defaults vs. the explicit max_length=512 truncation used during training account for the small swing) — treat 0.90 as the training-run number, not a guaranteed number for every inference path.

Repo

Full pipeline (data prep, weak labeling, audit, training script, Gradio demo): github.com/srijavuppala/jd-classifier-distilbert