qwen-darija-domain-adapted

Qwen2.5-3B Darija Domain Adaptation

A LoRA adapter that domain-adapts Qwen/Qwen2.5-3B-Instruct to Algerian Darija — Latin script, Arabic script, and French/English code-switching — via QLoRA continued pretraining.

This is stage 1 of a two-stage pipeline built for automated meeting-minutes generation on code-switched Algerian workplace speech. This end-to-end domain-adaptation approach outperformed a sequential translate-then-generate pipeline across every metric measured (ROUGE-L +29.7 pts, BERTScore F1 +3.0 pts, schema compliance +4.1 pts).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Chaima-KHENAFIF/qwen-darija-domain-adapted")
base_model.load_adapter("Chaima-KHENAFIF/qwen-darija-domain-adapted")

inputs = tokenizer("mochkil f la puce djezzy ma", return_tensors="pt").to(base_model.device)
output = base_model.generate(**inputs, max_new_tokens=30, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training

• QLoRA: 4-bit quantization (nf4, double quant), LoRA rank 16, alpha 32, applied to all attention and MLP projections
• Continued pretraining (causal LM objective, not instruction tuning) on a 10K-sample subset of a 182K-sample Algerian text corpus — limited to 10K by Kaggle's free-tier GPU quota
• 1 epoch, effective batch size 16, learning rate 2e-4

Results

Best validation perplexity: 25.78

Links

• Full code and training pipeline: GitHub repo