Downloads ยท 30 days
35
25% of all-time downloads
rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu
nllb-200-distilled-600M-int8-cpu is a translation model from rudrakshrakeshzodage. Use it when you need text moved from one language to another. The card lists the license as cc-by-nc-4.0.
[!NOTE] ๐ Live Web Demo: Try 100% offline in your browser at huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo
Downloads ยท 30 days
35
25% of all-time downloads
All-time downloads
141
Public
Repo size
1.7 GB
Likes
0
Public
Click a slice to open those files.
.bin1.7 GB ยท 99%
From the Hugging Face model README
[!NOTE] ๐ Live Web Demo: Try 100% offline in your browser at
huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo
[!NOTE] This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:
- FP32 (Full Precision / Base):
facebook/nllb-200-distilled-600M- FP16 (Half Precision):
facebook/nllb-200-distilled-600M- INT8 (Dynamic Quantization - CPU):
rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu(Current)- NF4 (4-bit GPU Quantization):
rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu- ONNX INT8 (Mobile & Edge Deployment):
rudrakshrakeshzodage/nllb-200-distilled-600M-onnx-int8(Current)
Base model: facebook/nllb-200-distilled-600M
Quantization: PyTorch INT8 Dynamic (torch.quantization.quantize_dynamic)
Languages: 200 (all NLLB-200 supported languages)


torch.nn.Linear inside Encoder & Decoder Self-Attention / FFN modules.qint8 (8-bit signed integers).| Metric | INT8 Dynamic (CPU) | FP16 Base (CPU) | Speedup / Savings |
|---|---|---|---|
| RAM Footprint | ~800 MB | ~2,400 MB | 66.7% RAM reduction |
| Avg Latency (Per Lang) | ~1,097 ms | ~3,950 ms | 3.6ร Faster |
| Throughput | 55 languages / min | ~15 languages / min | 3.6ร Higher |
| 200-Lang Benchmark Run | ~3.7 minutes | ~13.3 minutes | 9.6 min saved |
| Loop Anomalies | 0 โ | 1 (Central Kanuri) | Fully stable |
| BLEU Score (vs FP16) | ~42.0 | ~42.1 | < 0.1 BLEU loss |
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# Load base architecture
model = AutoModelForSeq2SeqLM.from_pretrained(
"facebook/nllb-200-distilled-600M"
)
# Apply PyTorch INT8 Dynamic Quantization
model = torch.quantization.quantize_dynamic(
model, {torch.nn.Linear}, dtype=torch.qint8
)
tokenizer = AutoTokenizer.from_pretrained(
"YOUR_USERNAME/nllb-200-distilled-600M-int8-cpu"
)
# Translate English to Hindi (hin_Deva)
inputs = tokenizer("Hello world, how are you?", return_tensors="pt")
translated_tokens = model.generate(
**inputs,
forced_bos_token_id=tokenizer.lang_code_to_id["hin_Deva"],
max_length=100
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])
Quantization research and benchmarking by RudrakshRakeshZodage.