qwen2.5-7b-urdu-v3

Qwen 2.5 7B Urdu (v3 LoRA adapter)

QLoRA fine-tune of Qwen/Qwen2.5-7B-Instruct specialised on Urdu instruction-following. Trained on ~69,000 instruction-response pairs covering Urdu script and Roman Urdu, with explicit code-mixed (Urdu-English) examples, Pakistani context, and — new in v3 — RAG-format (query, context, grounded-answer) triples so the model can actually use retrieved context.

This repo ships the LoRA adapter only (~154 MB). Apply on top of the base model with PEFT to use. See the Usage section below.

Built as a public build by a first-time fine-tuner. Full training pipeline + eval scripts + write-ups on GitHub. v3 story: WRITEUP_V3.md.

---

Headline result

79.5% pairwise preference vs base Qwen 2.5 7B Instruct across two independent LLM judges (Claude + GPT-5.3), on a 100-prompt hand-curated Urdu evaluation set — up from v2's 66%. All three v2 regressions (summarization, grammar, reasoning) recovered.

(Gemini 3.1 Pro, the 3rd v2 judge, was dropped for v3: its free tier caps at 20 requests/day, too few for the 100-prompt set. v3 was judged by the two cross-model judges that matter most for a stress test.)

Per-category win rate vs base

v3 vs v2, head-to-head: v3 wins only 43% of direct matchups — a rebalance, not a strict upgrade. It clearly wins summarization / grammar / code-mixed but trades away some translation / reasoning / QA. There is no free lunch in the data mix.

RAG (Urdu Wikipedia retrieval) — the honest result

v3 is trained on the exact {context}\n\nسوال:{query} surface the retrieval endpoint serves, which fixed RAG's structural failure:

• Chinese leakage: base Qwen + Urdu RAG context leaked Chinese on 45/100 answers → v3 + RAG leaks Chinese on 0/100. Now deployable.
• On factual prompts RAG corrects the model — e.g. Pakistan's largest province by area: plain v3 says "Punjab" (wrong), RAG says "Balochistan, 347,190 km²" from the article.
• But across the full 100-prompt set RAG beats plain v3 only 15.5% of the time: ~79 of the prompts are creative / grammar / reasoning where retrieved Wikipedia is pure noise.

RAG is a safe, deployable grounding tool for factual queries — not a blanket upgrade.

---

Quick start (Usage)

Recommended: load via Unsloth (the adapter was trained against Unsloth's 4-bit base — see adapter_config.json). Unsloth gives identical-base loading and ~2× inference speed.

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "TayyabManan/qwen2.5-7b-urdu-v3")
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "پاکستان کا قومی ترانہ کس نے لکھا؟"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
import torch
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9,
                         repetition_penalty=1.1, do_sample=True)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Using retrieved context (RAG)

v3 was trained on the exact surface below — reproduce it verbatim for grounded answers. Concatenate your retrieved chunks (newline-separated), then the question:

context = "\n\n".join(retrieved_chunks)   # your retriever's top-k passages
user = f"{context}\n\nسوال: {question}"
system = ("You are a helpful assistant. Use the provided context to answer in Urdu. "
          "If the context is insufficient, answer briefly from general knowledge.")
# recommended RAG decoding: temperature 0.3, top_p 0.9, repetition_penalty 1.3

Alternative: vanilla Transformers + PEFT

Works, but the adapter was calibrated against Unsloth's 4-bit base — expect minor numerical drift (no catastrophic failure).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(model, "TayyabManan/qwen2.5-7b-urdu-v3")
model.eval()

Recommended decoding

---

Training details

• Base model: Qwen/Qwen2.5-7B-Instruct (Apache 2.0)
• Method: QLoRA (4-bit NF4 base weights + LoRA)
• LoRA: r=16, alpha=32, targets = q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
• Effective batch: 16 (per-device 2 × grad_accum 8)
• Optimiser: AdamW 8-bit, lr=2e-4, cosine schedule, 3% warmup
• Epochs: 2 (same as v2 — 3 over-trains this data mix)
• Max sequence length: 4096 tokens (v2 was 2048; raised so RAG context fits without truncation)
• Precision: bfloat16 forward, 4-bit NF4 base
• Hardware: Single NVIDIA H100 80GB on Modal
• Wall-clock: 8,194 steps
• Final train loss: 0.5738
• Framework: Unsloth (2× throughput vs stock TRL on H100)
• Adapter size: ~154 MB
• Cost (single run): ~$5-6 on Modal H100

Training data (~69,000 examples total)

The v3 synthetic data was generated with GPT-4o-mini (~$0.63 total), deterministically checked (sentence counts, arithmetic digits, question presence), and human spot-checked (200 rows) before training — a review that caught two systemic generator bugs the automated tests missed (a 100%-broken reasoning template and confabulated RAG "noise" answers). Grammar gold answers are the original trusted sentence, never a model rewrite; reasoning answers are narrated over Python-computed numbers so the model never does math.

RAG-aware: unlike v2, the mix includes retrieval-format triples — a gold chunk, a gold chunk plus distractors, and "noise" (unrelated chunks where the correct move is to decline). This is what makes retrieved context in-distribution at inference time.

---

Limitations

• Reasoning is still the soft spot (53% vs base) — recovered from v2's 31% but the weakest category. Multi-step numeric word problems remain unreliable.
• RAG is not a blanket upgrade. It helps the minority of prompts that need a looked-up fact; on creative / grammar / reasoning prompts, prepended context is noise. Route retrieval at factual queries, not everything.
• v3 is a rebalance vs v2, not a strict superset. v3 loses to v2 on some translation / reasoning / QA prompts. Pick the version that matches your workload.
• English-leaning system-prompt sensitivity. Training used "You are a helpful assistant." (and the RAG system prompt above). Long or novel Urdu system prompts can destabilise generation.
• Repetition collapse on long Urdu prompts. Mitigate with max_new_tokens ≤ 512, repetition_penalty 1.3, temperature 0.3–0.5.
• Not for safety-critical use. Base-model refusal behaviours are preserved, not strengthened.
• Eval is pairwise preference, not absolute capability. A 79.5% win vs base does not mean 79.5% of answers are factually correct — many wins are stylistic / fluency-based.

---

Evaluation methodology

• Set: 100 hand-curated Urdu prompts across 8 categories (QA, summarization, translation, grammar, reasoning, creative, code explanation, code-mixed).
• Decoding parity: base and fine-tuned generate with identical decoding parameters.
• Judging: pairwise blinded preference, responses A/B-randomised per item to control position bias.
• Judges: 2 independent cross-model LLM judges (Claude + GPT-5.3).
• RAG eval: RAG-v3 vs plain-v3 judged over the same 100 prompts; plus a structural check for non-Urdu (Chinese) leakage.

Eval code: scripts/28_run_eval_v3.py

• scripts/18_run_judges.py
• scripts/17_aggregate_judges.py
• scripts/17b_aggregate_rag_vs_ft.py.

---

Acknowledgements

• Base model: Qwen team at Alibaba (Qwen 2.5 paper)
• Training framework: Unsloth by Daniel Han + team
• Retrieval: BAAI bge-m3 embedder + bge-reranker-v2-m3, Qdrant, rank-bm25
• Serverless compute: Modal
• Evaluation infrastructure: Haystack by deepset

---

Author

Muhammad Tayyab — MS Artificial Intelligence Engineering, COMSATS University Islamabad.

• GitHub: TayyabManan
• LinkedIn: linkedin.com/in/tayyabmanan
• Web: tayyabmanan.com

This is my first fine-tuning project end-to-end. Total project cost to date: ~$60.

---

Citation

@misc{tayyab2026qwen25urduv3,
  title  = {Qwen 2.5 7B Urdu (v3 LoRA adapter)},
  author = {Tayyab, Muhammad},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/TayyabManan/qwen2.5-7b-urdu-v3}
}

License

Apache 2.0. Inherits from Qwen 2.5 base.