av-job-relevance-setfit

AV Job Relevance Classifier (SetFit)

A few-shot binary classifier that predicts whether a job posting is relevant to autonomous vehicle (AV) software engineering, given its title and description. It's a distilled, zero-API-cost stand-in for the LLM-based relevance screening stage of the Autonomous Vehicle Job Profiles pipeline (Group 3) — instead of paying for a Groq/LLM call on every job, this model runs locally after being fine-tuned on a small set of LLM-labeled examples.

Model details

• Base model: sentence-transformers/all-MiniLM-L6-v2
• Method: SetFit — contrastive fine-tuning of the sentence encoder itself on the labeled seed, plus a logistic-regression classification head. This is a genuine few-shot technique (not a frozen-embedding linear probe): the encoder's own weights are updated, which is what makes it competitive on a small labeled set.
• Task: binary classification — is_av_relevant (True/False)
• Training data: job postings scraped from public ATS boards (Greenhouse, Lever, Ashby, etc.) for AV/robotics companies and general tech companies, labeled by an LLM (Groq, openai/gpt-oss-20b) as AV-relevant or not. See scrapers/service/llm/job_classifier.py in the source repo for the labeling prompt/logic.
• Not evaluated here: this repo does not currently publish a fixed held-out accuracy/precision/recall for this checkpoint — evaluation is done ad hoc via relevance_classifier_cli.py train --test-size <fraction> against whatever labeled seed exists at training time, and isn't tracked as a permanent artifact. Re-run that command against your own data if you need a number for a specific checkpoint.

Usage

from setfit import SetFitModel

model = SetFitModel.from_pretrained("husthunterpy01/av-job-relevance-setfit")
probs = model.predict_proba([
    "Senior Perception Engineer working on camera and LiDAR sensor fusion for autonomous driving.",
    "Staff Accountant handling month-end close and accounts payable.",
])
# probs[i][1] = P(AV-relevant)

Or via the source repo's wrapper, which also handles the local-weights -> Hub fallback automatically:

from scrapers.service.ml.setfit_classifier import SetFitRelevanceClassifier

classifier = SetFitRelevanceClassifier.load("data/job_classification/relevance_model_setfit")
classifier.predict_proba(["..."])

CLI (from the repo root):

python -m scrapers.utils.relevance_classifier_cli score \
  --input data/silver_export.jsonl --backend setfit

Intended use and limitations

This is an internal tool for a student capstone project's data pipeline, not a general-purpose job classifier. It:

• Is trained on a relatively small, English-language, tech-industry-skewed seed set built from a specific set of scraped companies — it will not generalize well outside that distribution (e.g. non-English postings, unrelated industries).
• Outputs a binary AV-relevance signal only; it does not assign the 9-category AV taxonomy (Perception, Planning, Control, etc.) — that's a separate LLM enrichment step (job_enricher.py) in the source pipeline.
• Was distilled from LLM labels, so it inherits whatever labeling errors or biases the LLM (and the prompt it was given) had.

Training your own version

See scrapers/README.md and scrapers/utils/relevance_classifier_cli.py in the source repository:

python -m scrapers.utils.relevance_classifier_cli train --backend setfit