multitask_model_fon_True_multiplicative

• Paper: FonMTL: Toward Building a Multi-Task Learning Model for Fon Language, accepted at WiNLP co-located at EMNLP 2023

• Official Github: https://github.com/bonaventuredossou/multitask_fon

• Build Multi-task Learning Model: For the shared layers (encoders) we used the following language model heads:

  • AfroLM-Large

    • AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages (Dossou et.al., EMNLP 2022)

  • XLMR-Large:

    • Unsupervised Cross-lingual Representation Learning at Scale (Conneau et.al., ACL 2020)

• Evaluation:

  • The goal primarily is to explore whether multitask learning improves performance on downstream tasks for Fon. We try two settings: (a) training only on Fon and evaluating on Fon, (b) training on all languages and evaluating on Fon. We evaluate the multi-task learning model on NER and POS tasks, and compare it with baselines (models finetuned and evaluated on single tasks)

How to get started

• Run the training: sbatch run.sh

This command will:

• Set up the environement
• Install required libraries: pip install -r requirements.txt -q
• Move to the code folder: cd code
• Run the training & evaluate: python run_train.py

NER Results

POS Results

Importance of Merging Representation Type

Model End-Points

• multitask_model_fon_False_multiplicative.bin is the MTL Fon Model which has been pre-trained on all MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.

• multitask_model_fon_True_multiplicative.bin is the MTL Fon Model which has been pre-trained only on Fon data from the MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.

How to run inference when you have the model

To run inference with the model(s), you can use the testing block defined in our MultitaskFON class.

TODO

• leverage the impact of the dynamic weighted average loss