distilbert-base-uncased-finetuned-FiNER

distilbert-base-uncased-finetuned-FiNER

This model is a fine-tuned version of distilbert/distilbert-base-uncased trained on a subset of the nlpaueb/finer-139 dataset. The subset is generated by filtering the dataset to contain only samples with at least one of the following NER tags:

• 'O',
• 'B-DebtInstrumentBasisSpreadOnVariableRate1',
• 'B-DebtInstrumentFaceAmount',
• 'B-LineOfCreditFacilityMaximumBorrowingCapacity',
• 'B-DebtInstrumentInterestRateStatedPercentage'

Then, it was fine-tuned to detect only the afforementioned 4 tags (plus other "O")

It achieves the following results on the evaluation set:

• Loss: 0.0336
• Precision: 0.9154
• Recall: 0.9327
• F1: 0.9240
• Accuracy: 0.9917

Model description

Model based on distilbert/distilbert-base-uncased with all default parameters.

Intended uses & limitations

The model published here was trained for demo purposes only.

Training and evaluation data

Original train/validation/test splits from nlpaueb/finer-139, after filtering for samples containing at least one of the following NER tags:

• 'O',
• 'B-DebtInstrumentBasisSpreadOnVariableRate1',
• 'B-DebtInstrumentFaceAmount',
• 'B-LineOfCreditFacilityMaximumBorrowingCapacity',
• 'B-DebtInstrumentInterestRateStatedPercentage'

Training procedure

Follow information here https://github.com/bodias/DistilBERT-FiNER

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 6

Training results

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2