nllb-200-distilled-600M-nf4-4bit-gpu

NLLB-200 Distilled 600M — NF4 4-bit GPU Quantized

[!NOTE] 🌐 Live Web Demo: Try 100% offline in your browser at huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo

This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:

• FP32 (Full Precision / Base): rudrakshrakeshzodage/nllb-200-distilled-600M-fp32
• FP16 (Half Precision): rudrakshrakeshzodage/nllb-200-distilled-600M-fp16
• INT8 (Dynamic Quantization - CPU): rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu
• NF4 (4-bit GPU Quantization): rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu
• ONNX INT8 (Mobile & Edge Deployment): rudrakshrakeshzodage/nllb-200-distilled-600M-onnx-int8 (Current)

Base model: facebook/nllb-200-distilled-600M
Quantization: NF4 4-bit via bitsandbytes
Languages: 200 (all NLLB-200 supported languages)

---

🏗️ Detailed Tokenizer & Model Architecture Pipeline

💾 Model Size Reduction Comparison (FP32 vs FP16 vs INT8 vs NF4 vs ONNX Mobile)

⚡ Hardware Benchmark: Laptop vs Mobile On-Device Inference

📈 200+ Language Performance Benchmark Chart

Quantization Architecture Details

• Linear Layers: Quantized to NormalFloat4 (NF4) format via bitsandbytes.
• Double Quantization: Enabled (quantizes quantization constants for minimal memory footprint).
• Compute Dtype: torch.float16 for execution on CUDA Tensor Cores.
• Embedding & LM Head: Maintained in FP16 for translation precision.

---

📊 Model Performance & Benchmarks

---

💻 Hardware Requirements

• GPU: NVIDIA GPU with 3 GB+ VRAM (GTX 1060 6GB minimum)
• RAM: 6 GB System Memory
• Software: CUDA 11.8+, bitsandbytes>=0.39.0, transformers>=4.28.0

---

🚀 Usage

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    "YOUR_USERNAME/nllb-200-distilled-600M-nf4-4bit-gpu",
    quantization_config=quant_config,
    device_map="cuda",
)

tokenizer = AutoTokenizer.from_pretrained(
    "YOUR_USERNAME/nllb-200-distilled-600M-nf4-4bit-gpu"
)

# Translate English to Hindi (hin_Deva)
inputs = tokenizer("Hello world, how are you?", return_tensors="pt").to("cuda")
translated_tokens = model.generate(
    **inputs, 
    forced_bos_token_id=tokenizer.lang_code_to_id["hin_Deva"], 
    max_length=100
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])

---

📜 Citation & Credits

Quantization research, engineering, and benchmarking developed by Rudraksh Rakesh Zodage:

• Hugging Face: rudrakshrakeshzodage
• GitHub: RudrakshRakeshZodage
• LinkedIn: rudraksh-zodage-
• Email: [email protected]

---

<!-- ## 🔍 Search Indexing & Metadata * **Developer:** Rudraksh Rakesh Zodage * **Project:** translation-model-quantization (model-quantization-lab) * **Hugging Face Profile:** [rudrakshrakeshzodage](https://huggingface.co/rudrakshrakeshzodage) * **GitHub Profile:** [RudrakshRakeshZodage](https://github.com/RudrakshRakeshZodage/) * **LinkedIn:** [rudraksh-zodage-](https://www.linkedin.com/in/rudraksh-zodage-/) * **Core Technologies:** No Language Left Behind (NLLB-200), Neural Machine Translation (NMT), post-training quantization, 4-bit NormalFloat (NF4), INT8 dynamic range, bitsandbytes, ONNX, PyTorch, LoRA, QLoRA fine-tuning, Model Compression, Edge AI deployment, Rudraksh Zodage. -->