Downloads · 30 days
0
tokenizer-mbart-25-enis
mideind/tokenizer-mbart-25-enis
tokenizer-mbart-25-enis is a translation model from mideind. Use it when you need text moved from one language to another. The card lists the license as mit.
This tokenizer is used for Mideind's mBART translation models. It is based on Facebooks mBART-25 SentencePiece model. A language token from the original model has been replaced with "isIS".
Access
Public
Updated Mar 28, 2022
Repo size
22.2 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json17.1 MB · 77%
At a glance
- Task
- Translation
- License
- mit
- Access
- Public
- Created
- Mar 28, 2022
- Updated
- Mar 28, 2022
- SHA
- bbe6246d
Model card
From the Hugging Face model README
mBART 25 SentencePiece tokenizer
This tokenizer is used for Mideind's mBART translation models. It is based on Facebooks mBART-25 SentencePiece model. A language token from the original model has been replaced with "is_IS".
Usage example (for debugging):
import sys
from transformers.models import mbart
MODEL_DIR = sys.argv[1]
tokenizer: mbart.MBartTokenizerFast = mbart.MBartTokenizerFast.from_pretrained(
MODEL_DIR, src_lang="en_XX"
)
is_lang_idx = tokenizer.convert_tokens_to_ids("is_IS")
model = mbart.MBartForConditionalGeneration.from_pretrained(MODEL_DIR)
test_sentence = "This is a test."
input_ids = tokenizer(test_sentence, return_tensors="pt")
print(input_ids)
outputs = model.generate(
**input_ids, decoder_start_token_id=is_lang_idx
)
print(outputs)
print(tokenizer.batch_decode(outputs))
- Task
- Translation
- License
- mit
- Languages
- is, en
- Created
- Mar 28, 2022
- Updated
- Mar 28, 2022