jobbert-base-cased-compdecs

Nesta, the UK's innovation agency, has been scraping online job adverts since 2021 and building algorithms to extract and structure information as part of the Open Jobs Observatory project.

Although we are unable to share the raw data openly, we aim to open source our models, algorithms and tools so that anyone can use them for their own research and analysis.

🖊️ Model description

This model is a fine-tuned version of jjzha/jobbert-base-cased. JobBERT is a continuously pre-trained bert-base-cased checkpoint on ~3.2M sentences from job postings.

It has been fine tuned with a classification head to binarily classify job advert sentences as being a company description or not.

The model was trained on 486 manually labelled company description sentences and 1000 non company description sentences less than 250 characters in length.

It achieves the following results on a held out test set 147 sentences:

• Accuracy: 0.92157

The code for training the model is in our ojd_daps_language_models repo, a central repository for fine-tuning transformer models on our database of scraped job adverts.

🖨️ Use

To use the model:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import pipeline

model = AutoModelForSequenceClassification.from_pretrained("nestauk/jobbert-base-cased-compdecs")
tokenizer = AutoTokenizer.from_pretrained("nestauk/jobbert-base-cased-compdecs")

comp_classifier = pipeline('text-classification', model=model, tokenizer=tokenizer)

An example use is as follows:

job_sent = "Would you like to join a major manufacturing company?"
comp_classifier(job_sent)

>> [{'label': 'LABEL_1', 'score': 0.9953641891479492}]

The intended use of this model is to extract company descriptions from online job adverts to use in downstream tasks such as mapping to Standardised Industrial Classification (SIC) codes.

⚖️ Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

⚖️ Training results

The fine-tuning metrics are as follows:

• eval_loss: 0.462236
• eval_runtime: 0.629300
• eval_samples_per_second: 233.582000
• eval_steps_per_second: 15.890000
• epoch: 10.000000
• perplexity: 1.590000

⚖️ Framework versions

• Transformers 4.32.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.4
• Tokenizers 0.13.3