llama-3.2-1b-urdu-en-qlora

llama-3.2-1b-urdu-en-qlora

QLoRA fine-tune of meta-llama/Llama-3.2-1B-Instruct for Urdu→English translation.

Training

• Dataset: kashif1085/urdu-en-translation-instruct (filtered subset of Helsinki-NLP/opus-100, en-ur)
• QLoRA: r=16, alpha=32, dropout=0.05
• 4-bit NF4 + double-quant, fp16 compute (T4 GPU)
• 3 epochs, lr=0.0002, cosine schedule
• Effective batch size: 2 × 4 = 8
• Hardware: Colab T4 (free tier), trained with Unsloth (~2x speedup)
• Total training time: ~63 minutes

Evaluation (n=200)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.2-1B-Instruct')
tok  = AutoTokenizer.from_pretrained('meta-llama/Llama-3.2-1B-Instruct')
model = PeftModel.from_pretrained(base, 'kashif1085/llama-3.2-1b-urdu-en-qlora')

Limitations

• Trained on a small slice (8000 pairs) — not a production translator.
• Biased toward the news-domain style of opus-100.
• Portfolio piece. Not for real-world deployment.
• BLEU 10 is well below production NMT systems (Google Translate ~30-40 BLEU on this language pair). The +154% relative improvement over base demonstrates the technique works at small scale.