Fine-Tashkeel

Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization

Table of Contents

• Introduction
• Models
  • ByT5
    • Model Description
• Benchmarks
• Citation
• Contact

Introduction

Most of previous work on learning diacritization of the Arabic language relied on training models from scratch. In this paper, we investigate how to leverage pre-trained language models to learn diacritization. We finetune token-free pre-trained multilingual models (ByT5) to learn to predict and insert missing diacritics in Arabic text, a complex task that requires understanding the sentence semantics and the morphological structure of the tokens. We show that we can achieve state-of-the-art on the diacritization task with minimal amount of training and no feature engineering, reducing WER by 40%. We release our finetuned models for the greater benefit of the researchers in the community.

Model Description

The ByT5 model, distinguished by its innovative token-free architecture, directly processes raw text to adeptly navigate diverse languages and linguistic nuances. Pre-trained on a comprehensive text corpus mc4, ByT5 excels in understanding and generating text, making it versatile for various NLP tasks. We have further enhanced its capabilities by fine-tuning it on a Tashkeela data set for 13,000 steps, significantly refining its performance in restoring the diacritical marks for Arabic.

Benchmarks

Note: This model has been trained specifically for use with Classical Arabic.

Our model attained a Diacritics Error Rate (DER) of 0.95 and a Word Error Rate (WER) of 2.49.

Code sample to use the model.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import pandas as pd


if __name__ == "__main__":

  text = "كيف الحال"
  
  model_name = "basharalrfooh/Fine-Tashkeel" 
  
  tokenizer = AutoTokenizer.from_pretrained(model_name)
  model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
  
  input_ids = tokenizer(text, return_tensors="pt").input_ids
  outputs = model.generate(input_ids, max_new_tokens=128)
  decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
  print("Generated output:", decoded_output)

Citation

@misc{alrfooh2023finetashkeel,
      title={Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization}, 
      author={Bashar Al-Rfooh and Gheith Abandah and Rami Al-Rfou},
      year={2023},
      eprint={2303.14588},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contact

[email protected]