Downloads · 30 days
32
10% of all-time downloads
cartesinus/iva_mt_wslot-m2m100_1.2B-en-pl
iva_mt_wslot-m2m100_1.2B-en-pl is a machine learning model from cartesinus. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
32
10% of all-time downloads
All-time downloads
336
Public
Parameters
1.2B
54.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.bin5 GB · 50%
From the Hugging Face model README
This model is a fine-tuned version of facebook/m2m100_1.2B on the iva_mt_wslot dataset. There is also smaller version of this model here: cartesinus/iva_mt_wslot-m2m100_418M-en-pl. This model (1.2B) achieves the following results:
For reference BLEU for baseline m2m100-418M (plain text) was 21.9468 and for m2m100-1.2B was (result in preparation). Second result (BLEU with slots) is when tags are treated as "normal" words in sentence. Therefore that result might be a bit misleading. Please refer to plain text results if you are not sure how to interpret them.
For reference WMT20 submission systems in en-pl direction had between 25 and 30 BLEU
BLEU on the evaluation set (same as in below table 'Training results'): 62.4604
On the training set (to see how it adjusted to train):
BLEU was measured with sacreBLEU library.
Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, imperatives with a lot of name entities (slots) and particular vocabulary (for example settings name). It can be observed in above results where WMT results are very low while in-domain test is very high.
This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it will be ok. If you will translate out-of-domain sentenences (such as for example News, Medical) that are not very similar then results will drop significantly.
One last thing that needs to be mentioned is that BLEU is not particulary good metric to evaluate IVA sentences due to their length and it should be evalued with other metrices (e.g. GLEU).
First please make sure to install pip install transformers. First download model:
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import torch
def translate(input_text, lang):
input_ids = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
model_name = "cartesinus/iva_mt_wslot-m2m100_1.2B-en-pl"
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="pl")
model = M2M100ForConditionalGeneration.from_pretrained(model_name)
Then you can translate either plan text like this:
print(translate("set the temperature on my thermostat", "pl"))
or you can translate with slot annotations that will be restored in tgt language:
print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "pl")) #translation: obudź mnie o <a>piątej rano<a> <b>w tym tygodniu<b>
Limitations of translation with slot transfer:
| Corpus | Train | Dev | Test |
|---|---|---|---|
| Massive 1.1 | 11514 | 2033 | 2974 |
| Leyzer 0.2.0 | 3974 | 701 | 1380 |
| OpenSubtitles from OPUS | 2329 | 411 | 500 |
| KDE from OPUS | 1154 | 241 | 241 |
| CCMatrix from Opus | 1096 | 232 | 237 |
| Ubuntu from OPUS | 281 | 60 | 59 |
| Gnome from OPUS | 14 | 3 | 3 |
| total | 20362 | 3681 | 5394 |
| Training Loss | Epoch | Step | Validation Loss | Bleu | Gen Len |
|---|---|---|---|---|---|
| 0.2744 | 1.0 | 5091 | 0.2555 | 58.5119 | 21.0728 |
| 0.1829 | 2.0 | 10182 | 0.2475 | 59.7364 | 21.0769 |
| 0.1124 | 3.0 | 15273 | 0.2499 | 61.3552 | 21.06 |
| 0.0783 | 4.0 | 20364 | 0.2597 | 61.6618 | 21.2402 |
| 0.0496 | 5.0 | 25455 | 0.2698 | 62.1942 | 21.2901 |
| 0.0318 | 6.0 | 30546 | 0.2798 | 61.9068 | 21.3399 |
| 0.0204 | 7.0 | 35637 | 0.2893 | 61.7753 | 21.3102 |
| 0.0138 | 8.0 | 40728 | 0.2979 | 62.3925 | 21.3238 |
| 0.009 | 9.0 | 45819 | 0.3034 | 62.4942 | 21.2516 |
| 0.0058 | 10.0 | 50910 | 0.3082 | 62.4604 | 21.2847 |