TigXLNet

Transferring Monolingual Model to Low-Resource Language: The Case Of Tigrinya:

Proposed Method:

<img src="data/proposed.png" height = "330" width ="760" >

The proposed method transfers a mono-lingual Transformer model into new target language at lexical level by learning new token embeddings. All implementation in this repo uses XLNet as a source Transformer model, however, other Transformer models can also be used similarly.

Main files:

All files are IPython Notebook files which can be excuted simply in Google Colab.

• train.ipynb : Fine-tunes XLNet (mono-lingual transformer) on new target language (Tigrinya) sentiment analysis dataset.

• test.ipynb : Evaluates the fine-tuned model on test data.

• token_embeddings.ipynb : Trains a word2vec token embeddings for Tigrinya language.

• process_Tigrinya_comments.ipynb : Extracts Tigrinya comments from mixed language contents.

• extract_YouTube_comments.ipynb : Downloads available comments from a YouTube channel ID.

• auto_labelling.ipynb : Automatically labels Tigrinya comments in to positive or negative sentiments based on Emoji's sentiment.

Tigrinya Tokenizer:

A sentencepiece based tokenizer for Tigrinya has been released to the public and can be accessed as in the following:

 from transformers import AutoTokenizer
 tokenizer = AutoTokenizer.from_pretrained("abryee/TigXLNet")
 tokenizer.tokenize("ዋዋዋው እዛ ፍሊም ካብተን ዘድንቀን ሓንቲ ኢያ ሞ ብጣዕሚ ኢና ነመስግን ሓንቲ ክብላ ደልየ ዘሎኹ ሓደራኣኹም ኣብ ጊዜኹም ተረክቡ")

TigXLNet:

A new general purpose transformer model for low-resource language Tigrinya is also released to the public and be accessed as in the following:

from transformers import AutoConfig, AutoModel
config = AutoConfig.from_pretrained("abryee/TigXLNet")
config.d_head = 64
model = AutoModel.from_pretrained("abryee/TigXLNet", config=config)

Evaluation:

The proposed method is evaluated using two datasets:

• A newly created sentiment analysis dataset for low-resource language (Tigriyna).

<table> <tr> <td> <table> <thead> <tr> <th>_Models</th> <th>_{Configuration}</th> <th>_F1-Score</th> </tr> </thead> <tbody> <tr> <td rowspan=3>_BERT</td> <td rowspan=1>_{+Frozen BERT weights}</td> <td>_54.91</td> </tr> <tr> <td rowspan=1>_{+Random embeddings}</td> <td>_74.26</td> </tr> <tr> <td rowspan=1>_{+Frozen token embeddings}</td> <td>_76.35</td> </tr> <tr> <td rowspan=3>_mBERT</td> <td rowspan=1>_{+Frozen mBERT weights}</td> <td>_57.32</td> </tr> <tr> <td rowspan=1>_{+Random embeddings}</td> <td>_76.01</td> </tr> <tr> <td rowspan=1>_{+Frozen token embeddings}</td> <td>_77.51</td> </tr> <tr> <td rowspan=3>_XLNet</td> <td rowspan=1>_{+Frozen XLNet weights}</td> <td><strong>_68.14</strong></td> </tr> <tr> <td rowspan=1>_{+Random embeddings}</td> <td><strong>_77.83</strong></td> </tr> <tr> <td rowspan=1>_{+Frozen token embeddings}</td> <td><strong>_81.62</strong></td> </tr> </tbody> </table> </td> <td><img src="data/effect_of_dataset_size.png" alt="3" width = 480px height = 280px></td> </tr> </table>

• Cross-lingual Sentiment dataset (CLS).

<table> <thead> <tr> <th rowspan=2>_Models</th> <th rowspan=1 colspan=3>_English</th> <th rowspan=1 colspan=3>_German</th> <th rowspan=1 colspan=3>_French</th> <th rowspan=1 colspan=3>_Japanese</th> <th rowspan=2>_Average</th> </tr> <tr> <th colspan=1>_Books</th> <th colspan=1>_DVD</th> <th colspan=1>_Music</th> <th colspan=1>_Books</th> <th colspan=1>_DVD</th> <th colspan=1>_Music</th> <th colspan=1>_Books</th> <th colspan=1>_DVD</th> <th colspan=1>_Music</th> <th colspan=1>_Books</th> <th colspan=1>_DVD</th> <th colspan=1>_Music</th> </tr> </thead> <tbody> <tr> <td colspan=1>_XLNet</td> <td colspan=1>_{<strong>92.90</strong>}</td> <td colspan=1>_{<strong>93.31</strong>}</td> <td colspan=1>_{<strong>92.02</strong>}</td> <td colspan=1>_85.23</td> <td colspan=1>_83.30</td> <td colspan=1>_83.89</td> <td colspan=1>_73.05</td> <td colspan=1>_69.80</td> <td colspan=1>_70.12</td> <td colspan=1>_83.20</td> <td colspan=1>_{<strong>86.07</strong>}</td> <td colspan=1>_85.24</td> <td colspan=1>_83.08</td> </tr> <tr> <td colspan=1>_mBERT</td> <td colspan=1>_92.78</td> <td colspan=1>_90.30</td> <td colspan=1>_91.88</td> <td colspan=1>_{<strong>88.65</strong>}</td> <td colspan=1>_{<strong>85.85</strong>}</td> <td colspan=1>_{<strong>90.38</strong>}</td> <td colspan=1>_{<strong>91.09</strong>}</td> <td colspan=1>_{<strong>88.57</strong>}</td> <td colspan=1>_{<strong>93.67</strong>}</td> <td colspan=1>_{<strong>84.35</strong>}</td> <td colspan=1>_81.77</td> <td colspan=1>_{<strong>87.53</strong>}</td> <td colspan=1>_{<strong>88.90</strong>}</td> </tr> </tbody> </table>

Dataset used for this paper:

We have constructed new sentiment analysis dataset for Tigrinya language and it can be found in the zip file (Tigrinya Sentiment Analysis Dataset)

Citing our paper:

Our paper can be accessed from ArXiv link, and please consider citing our work.

 @misc{tela2020transferring,
      title={Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya},
      author={Abrhalei Tela and Abraham Woubie and Ville Hautamaki},
      year={2020},
      eprint={2006.07698},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
  }

Any questions, comments, feedback is appreciated! And can be forwarded to the following email: [email protected]