Downloads · 30 days
0
tartuNLP/smugri3_14-finno-ugric-nmt
smugri3_14-finno-ugric-nmt is a translation model from tartuNLP. Use it when you need text moved from one language to another. It is set up for fairseq. The card lists the license as cc-by-4.0.
The TartuNLP Multilingual Neural Machine Translation model for low-resource Finno-Ugric languages. The model can translate in 702 directions, between 27 languages.
Downloads · 30 days
0
Access
Public
Updated Apr 16, 2024
Repo size
9.6 GB
Likes
3
Public
Click a slice to open those files.
.pt9.6 GB · 100%
From the Hugging Face model README
The TartuNLP Multilingual Neural Machine Translation model for low-resource Finno-Ugric languages. The model can translate in 702 directions, between 27 languages.
The model can be tested in our web demo.
To use this model for translation tasks, you will need to utilize the Fairseq v0.12.2.
Bash script example:
# Define target and source languages
src_lang="eng_Latn"
tgt_lang="kpv_Cyrl"
# Directories and paths
model_path=./smugri3_14-finno-ugric-nmt
checkpoint_path=${model_path}/smugri3_14.pt
sp_path=${model_path}/flores200_sacrebleu_tokenizer_spm.ext.model
dictionary_path=${model_path}/nllb_model_dict.ext.txt
# Language settings for fairseq
nllb_langs="eng_Latn,est_Latn,fin_Latn,hun_Latn,lvs_Latn,nob_Latn,rus_Cyrl"
new_langs="kca_Cyrl,koi_Cyrl,kpv_Cyrl,krl_Latn,liv_Latn,lud_Latn,mdf_Cyrl,mhr_Cyrl,mns_Cyrl,mrj_Cyrl,myv_Cyrl,olo_Latn,sma_Latn,sme_Latn,smj_Latn,smn_Latn,sms_Latn,udm_Cyrl,vep_Latn,vro_Latn"
# Start fairseq-interactive in interactive mode
fairseq-interactive ${model_path} \
-s ${src_lang} -t ${tgt_lang} \
--path ${checkpoint_path} --max-tokens 20000 --buffer-size 1 \
--beam 4 --lenpen 1.0 \
--bpe sentencepiece \
--remove-bpe \
--lang-tok-style multilingual \
--sentencepiece-model ${sp_path} \
--fixed-dictionary ${dictionary_path} \
--task translation_multi_simple_epoch \
--decoder-langtok --encoder-langtok src \
--lang-pairs ${src_lang}-${tgt_lang} \
--langs "${nllb_langs},${new_langs}" \
--cpu
Average:
| to-lang | bleu | chrf | chrf++ |
|---|---|---|---|
| ru | 24.82 | 51.81 | 49.08 |
| en | 28.24 | 55.91 | 53.73 |
| et | 18.66 | 51.72 | 47.69 |
| fi | 15.45 | 50.04 | 45.38 |
| hun | 16.73 | 47.38 | 44.19 |
| lv | 18.15 | 49.04 | 45.54 |
| nob | 14.43 | 45.64 | 42.29 |
| kpv | 10.73 | 42.34 | 38.50 |
| liv | 5.16 | 29.95 | 27.28 |
| mdf | 5.27 | 37.66 | 32.99 |
| mhr | 8.51 | 43.42 | 38.76 |
| mns | 2.45 | 27.75 | 24.03 |
| mrj | 7.30 | 40.81 | 36.40 |
| myv | 4.72 | 38.74 | 33.80 |
| olo | 4.63 | 34.43 | 30.00 |
| udm | 7.50 | 40.07 | 35.72 |
| krl | 9.39 | 42.74 | 38.24 |
| vro | 8.64 | 39.89 | 35.97 |
| vep | 6.73 | 38.15 | 33.91 |
| lud | 3.11 | 31.50 | 27.30 |
Evaluated with Smugri Flores testset.