distilbert-multi-finetuned-for-xqua-on-tydiqa

DistilBERT multilingual fine-tuned on TydiQA (GoldP task) dataset for multilingual Q&A 😛🌍❓

Details of the language model

distilbert-base-multilingual-cased

Details of the Tydi QA dataset

TyDi QA contains 200k human-annotated question-answer pairs in 11 Typologically Diverse languages, written without seeing the answer and without the use of translation, and is designed for the training and evaluation of automatic question answering systems. This repository provides evaluation code and a baseline system for the dataset. https://ai.google.com/research/tydiqa

Details of the downstream task (Gold Passage or GoldP aka the secondary task)

Given a passage that is guaranteed to contain the answer, predict the single contiguous span of characters that answers the question. the gold passage task differs from the primary task in several ways:

• only the gold answer passage is provided rather than the entire Wikipedia article;
• unanswerable questions have been discarded, similar to MLQA and XQuAD;
• we evaluate with the SQuAD 1.1 metrics like XQuAD; and
• Thai and Japanese are removed since the lack of whitespace breaks some tools.

Model training 💪🏋️‍

The model was fine-tuned on a Tesla P100 GPU and 25GB of RAM. The script is the following:

python transformers/examples/question-answering/run_squad.py \
  --model_type distilbert \
  --model_name_or_path distilbert-base-multilingual-cased \
  --do_train \
  --do_eval \
  --train_file /path/to/dataset/train.json \
  --predict_file /path/to/dataset/dev.json \
  --per_gpu_train_batch_size 24 \
  --per_gpu_eval_batch_size 24 \
  --learning_rate 3e-5 \
  --num_train_epochs 5 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --overwrite_output_dir \
  --save_steps 1000 \
  --threads 400

Global Results (dev set) 📝

Specific Results (per language) 🌍📝

Similar models

You can also try bert-multi-cased-finedtuned-xquad-tydiqa-goldp that achieves F1 = 82.16 and EM = 71.06 (And of course better marks per language).

Created by Manuel Romero/@mrm8488

Made with <span style="color: #e25555;">♥</span> in Spain