AraElectra-Arabic-SQuADv2-QA

AraElectra for Question Answering on Arabic-SQuADv2

This is the AraElectra model, fine-tuned using the Arabic-SQuADv2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering. with help of AraElectra Classifier to predicted unanswerable question.

Overview

Language model: AraElectra <br> Language: Arabic <br> Downstream-task: Extractive QA
Training data: Arabic-SQuADv2.0
Eval data: Arabic-SQuADv2.0 <br> Test data: Arabic-SQuADv2.0 <br> Code: See More Info on Github
Infrastructure: 1x Tesla K80

Hyperparameters

batch_size = 8
n_epochs = 4
base_LM_model = "AraElectra"
learning_rate = 3e-5
optimizer = AdamW
padding = dynamic

Online Demo on Arabic Wikipedia and User Provided Contexts

See model in action hosted on streamlit

Usage

For best results use the AraBert preprocessor by aub-mind

from transformers import ElectraForQuestionAnswering, ElectraForSequenceClassification, AutoTokenizer, pipeline
from preprocess import ArabertPreprocessor
prep_object = ArabertPreprocessor("araelectra-base-discriminator")
question = prep_object('ما هي جامعة الدول العربية ؟')
context = prep_object('''
جامعة الدول العربية هيمنظمة إقليمية تضم دولاً عربية في آسيا وأفريقيا.
ينص ميثاقها على التنسيق بين الدول الأعضاء في الشؤون الاقتصادية، ومن ضمنها العلاقات التجارية الاتصالات، العلاقات الثقافية، الجنسيات ووثائق وأذونات السفر والعلاقات الاجتماعية والصحة. المقر الدائم لجامعة الدول العربية يقع في القاهرة، عاصمة مصر (تونس من 1979 إلى 1990). 
''')
# a) Get predictions
qa_modelname = 'ZeyadAhmed/AraElectra-Arabic-SQuADv2-QA'
cls_modelname = 'ZeyadAhmed/AraElectra-Arabic-SQuADv2-CLS'
qa_pipe = pipeline('question-answering', model=qa_modelname, tokenizer=qa_modelname)
QA_input = {
    'question': question,
    'context': context
}
CLS_input = {
    'text': question,
    'text_pair': context
}
qa_res = qa_pipe(QA_input)
cls_res = cls_pipe(CLS_iput)
threshold = 0.5 #hyperparameter can be tweaked
## note classification results label0 probability it can be answered label1 probability can't be answered 
## if label1 probability > threshold then consider the output of qa_res is empty string else take the qa_res
# b) Load model & tokenizer
qa_model = ElectraForQuestionAnswering.from_pretrained(qa_modelname)
cls_model = ElectraForSequenceClassification.from_pretrained(cls_modelname)
tokenizer = AutoTokenizer.from_pretrained(qa_modelname)

Performance

Evaluated on the Arabic-SQuAD 2.0 test set with the official eval script except changing in the preprocessing a little to fit the arabic language the modified eval script.

"exact": 65.11555277951281,
"f1": 71.49042547237256,,

"total": 9606,
"HasAns_exact": 56.14535768645358,
"HasAns_f1": 67.79623803036668,
"HasAns_total": 5256,
"NoAns_exact": 75.95402298850574,
"NoAns_f1": 75.95402298850574,
"NoAns_total": 4350