nllb-600M-finetune-en-kha

NLLB 600M for Khasi

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = "ahlad/nllb-600M-finetune-en-kha"
tokenizer = AutoTokenizer.from_pretrained(model_name, src_lang="vie_Latn")
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

article = "Kata ka dei ka bos ."
inputs = tokenizer(article, return_tensors="pt")

translated_tokens = model.generate(
    **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("eng_Latn"), max_length=30
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])

Pipeline

This is the preferred method for translating a large number of sentences when used in conjunction with a Hugging Face Dataset.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline
import torch


model_name = "ahlad/nllb-600M-finetune-en-kha"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

translator_nllb = pipeline(
    "translation",
    model=model,
    tokenizer=tokenizer,
    src_lang="vie_Latn",
    tgt_lang="eng_Latn",
    max_length=128,
    device=0 if torch.cuda.is_available() else -1,
)