nllb-200-600M-lez-rus-azj

Neural machine translation system for Lezgian, Russian and Azerbaijani languages

We release the first neural machine translation system for translation between Russian, Azerbaijani and the endangered Lezgian languages, as well as monolingual and parallel datasets collected and aligned for training and evaluating the system.

Get Usage

Russian-Lezgian example

from transformers import NllbTokenizer, AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("AlidarAsvarov/nllb-200-600M-lez-rus-azj")
tokenizer = NllbTokenizer.from_pretrained("AlidarAsvarov/nllb-200-600M-lez-rus-azj", )

model.eval()

tokenizer.src_lang = 'rus_Cyrl'
inputs = tokenizer(
    'Футбольная ассоциации Англии рассчитывала провести финал кубка на перестроенном стадионе "Уэмбли"', 
    return_tensors='pt',
    padding=True,
    truncation=True,
    max_length=40
)

tokenizer.tgt_lang = 'lez_Cyrl'
tokenizer.batch_decode(
    model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(tokenizer.tgt_lang),
        max_new_tokens=int(32 + 3 * inputs.input_ids.shape[1]),
        num_beams=5
    ),
    skip_special_tokens=True
)

Azerbaijani-Lezgian example

from transformers import NllbTokenizer, AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("AlidarAsvarov/nllb-200-600M-lez-rus-azj")
tokenizer = NllbTokenizer.from_pretrained("AlidarAsvarov/nllb-200-600M-lez-rus-azj", )

model.eval()

tokenizer.src_lang = 'azj_Latn'
inputs = tokenizer(
    '1741-ci ilin payızında Nadir s ̧ah Mu ̈s ̧ku ̈r lәzgilәrinә hu ̈cum edib onlara divan tutur.', 
    return_tensors='pt',
    padding=True,
    truncation=True,
    max_length=102
)

tokenizer.tgt_lang = 'lez_Cyrl'
tokenizer.batch_decode(
    model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(tokenizer.tgt_lang),
        max_new_tokens=int(32 + 3 * inputs.input_ids.shape[1]),
        num_beams=5
    ),
    skip_special_tokens=True
)

Quality

BLEU score

ChrF score

Random examples of translation from Russian to Lezgian:

Citation

If you use that results in your research, please cite our paper:

@misc{asvarov2024neuralmachinetranslationlezgian,
      title={Neural machine translation system for Lezgian, Russian and Azerbaijani languages}, 
      author={Alidar Asvarov and Andrey Grabovoy},
      year={2024},
      eprint={2410.05472},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.05472}, 
}