nllb-600m-ct2-int8

NLLB-200 Distilled 600M (CTranslate2 INT8)

This repository contains the Facebook NLLB-200 Distilled 600M model converted to the CTranslate2 format using INT8 quantization.

Conversion

ct2-transformers-converter \
    --model facebook/nllb-200-distilled-600M \
    --output_dir nllb-600m-ct2 \
    --quantization int8

How to use?

import ctranslate2
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download

REPO_ID = "pulkitchowdry/nllb-600m-ct2-int8"

# Download model from Hugging Face
model_dir = snapshot_download(repo_id=REPO_ID)

print("Model downloaded to:", model_dir)

# Load translator
translator = ctranslate2.Translator(
    model_dir,
    device="cpu"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_dir, fix_mistral_regex=True)

src_lang = "eng_Latn"
tgt_lang = "fra_Latn"

text = "Hello, how are you today?"

tokenizer.src_lang = src_lang

# Tokenize
input_ids = tokenizer(text).input_ids
tokens = tokenizer.convert_ids_to_tokens(input_ids)

# Translate
results = translator.translate_batch(
    [tokens],
    target_prefix=[[tgt_lang]]
)

output_tokens = results[0].hypotheses[0]

translation = tokenizer.decode(
    tokenizer.convert_tokens_to_ids(output_tokens),
    skip_special_tokens=True,
)

print("Input:", text)
print("Translation:", translation)