job-ner-models

Job NER Pipeline Models

This repository contains two pretrained NER models for job-related text processing:

1. Job Title NER (title-distilbert-job-ner/)
   Detects entities in job titles.

2. Job Description NER (jd-distilbert-job-ner/)
   Detects entities in full job descriptions.

Both models include:

• model.safetensors (weights)
• config.json
• tokenizer.json
• tokenizer_config.json
• special_tokens_map.json

Repository Structure

README.md title_ner/ model.safetensors config.json tokenizer.json tokenizer_config.json special_tokens_map.json desc_ner/ model.safetensors config.json tokenizer.json tokenizer_config.json special_tokens_map.json

Usage

Install required packages:

pip install transformers huggingface_hub

Load models in Python:

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer, AutoModelForTokenClassification

# Download full repo
repo_dir = snapshot_download("username/job-ner-models")

# Job Title NER
title_path = f"{repo_dir}/title-distilbert-job-ner"
title_model = AutoModelForTokenClassification.from_pretrained(title_path)
title_tokenizer = AutoTokenizer.from_pretrained(title_path)

# Job Description NER
desc_path = f"{repo_dir}/jd-distilbert-job-ner"
desc_model = AutoModelForTokenClassification.from_pretrained(desc_path)
desc_tokenizer = AutoTokenizer.from_pretrained(desc_path)

You can now use these models in your pipeline for entity recognition on job titles and descriptions.

---

Notes

• Both models are required for the full pipeline.
• Do not modify the folder names; the pipeline expects title-distilbert-job-ner and jd-distilbert-job-ner.
• Large files (.safetensors) are tracked using Git LFS. Ensure you have it installed:

git lfs install