AraRoBERTa-KU

AraRoBERTa-KU

AraRoBERTa-KU is a RoBERTa-base model pre-trained from scratch on Kuwaiti Arabic text. It is one of seven mono-dialectal models introduced in Weakly and Semi-Supervised Learning for Arabic Text Classification using Monodialectal Language Models (WANLP 2022).

Usage

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="reemalyami/AraRoBERTa-KU")
fill_mask("الجو اليوم <mask> وايد")

For classification, load the model with AutoModelForSequenceClassification and fine-tune it as usual. During pre-training, tweets were normalized (Arabic letter normalization, with digits and character elongation removed), so applying similar preprocessing to your input may help.

Results

Binary dialect identification (F1) on manually annotated text plus NADI 2020 data:

With the paper's semi-supervised method, F1 is 0.89. With weak supervision from dialect dictionaries, it is 0.61.

The AraRoBERTa Family

Citation

@inproceedings{alyami-al-zaidy-2022-weakly,
    title = "Weakly and Semi-Supervised Learning for {A}rabic Text Classification using Monodialectal Language Models",
    author = "AlYami, Reem  and Al-Zaidy, Rabah",
    booktitle = "Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.wanlp-1.24",
    pages = "260--272",
}

Contact: Reem AlYami · LinkedIn · [email protected]