distilbert-base-nl-cased-finetuned-squad

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

distilbert-base-nl-cased-finetuned-squad

This model is a fine-tuned version of Geotrend/distilbert-base-nl-cased on the Dutch Squad V2 Dataset dataset [1], specifically tailored for the Question Answer task. It achieves the following results on the evaluation set:

• Loss: 1.2834

Model description

The base model, distilbert-base-nl-cased, is a smaller version of distilbert-base-multilingual-cased, designed to handle a custom number of languages (only Dutch in this case) while preserving the original model's accuracy. It is based on the principles outlined in the paper "Load What You Need: Smaller Versions of Multilingual BERT" by Abdaoui, Pradel, and Sigel (2020) [2].

Intended uses & limitations

This fine-tuned model is optimized for Dutch Question Answering tasks. While it may perform well on similar tasks in other languages, its primary strength lies in extracting answers from Dutch-language contexts. Users are encouraged to consider the model's specific training focus when applying it to different language or task scenarios.

Training and evaluation data

The model was trained on the Dutch Squad V2.0 Dataset, a machine-translated version of the original SQuAD v2.0 dataset. The statistics for both datasets are as follows:

Statistics

|                         | SQuAD v2.0       | Dutch SQuAD v2.0 |
|-------------------------|------------------|------------------|
| **Train**               |                  |                  |
| Total examples          |   130,319        |   95,054         |
| Positive examples       |   86,821         |   53,376         |
| Negative examples       |   43,498         |   41,768         |
| **Development**         |                  |                  |
| Total examples          |   11,873         |   9,294          |
| Positive examples       |   5,928          |   3,588          |
| Negative examples       |   5,945          |   5,706          |

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.4685        | 1.0   | 6229  | 1.2709          |
| 1.1882        | 2.0   | 12458 | 1.1931          |
| 0.9488        | 3.0   | 18687 | 1.2834          |

Raw Results

{
'exact': 59.479233914353344,
 'f1': 62.56163022484813,
 'total': 9294,
 'HasAns_exact': 38.405797101449274,
 'HasAns_f1': 46.390131357228995,
 'HasAns_total': 3588,
 'NoAns_exact': 72.73045916579039,
 'NoAns_f1': 72.73045916579039,
 'NoAns_total': 5706,
 'best_exact': 61.58812136862492,
 'best_exact_thresh': 0.0,
 'best_f1': 63.337535221120724,
 'best_f1_thresh': 0.0
}

Model Usage

To use this model, you can follow the example below:

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="tclungu/distilbert-base-nl-cased-finetuned-squad",
    tokenizer="tclungu/distilbert-base-nl-cased-finetuned-squad"
)

qa_pipeline({
    'context': "Amsterdam is de hoofdstad en de dichtstbevolkte stad van Nederland.",
    'question': "Wat is de hoofdstad van Nederland?"})

Output

{'score': 0.9984413385391235, 'start': 0, 'end': 9, 'answer': 'Amsterdam'}

Framework versions

• Transformers 4.33.3
• Pytorch 2.0.1
• Datasets 2.14.5
• Tokenizers 0.13.3

References

[1] Rouws, N. J., Vakulenko, S., & Katrenko, S. (2022). Dutch squad and ensemble learning for question answering from labour agreements. In Artificial Intelligence and Machine Learning: 33rd Benelux Conference on Artificial Intelligence, BNAIC/Benelearn 2021, Esch-sur-Alzette, Luxembourg, November 10–12, 2021, Revised Selected Papers 33 (pp. 155-169). Springer International Publishing.
[2] Abdaoui, A., Pradel, C., & Sigel, G. (2020). Load what you need: Smaller versions of multilingual bert. arXiv preprint arXiv:2010.05609.