KhulasaAI

KhulasaAI — Arabic Text Summarization (QLoRA Adapter)

A LoRA adapter for Qwen2.5-0.5B-Instruct fine-tuned on Arabic text summarization via Knowledge Distillation from Qwen2.5-7B-Instruct.

Model Description

Training Details

Dataset

• Source: Arabic Wikipedia (wikimedia/wikipedia, 20231101.ar)
• Size: 5,000 articles randomly sampled (seed=42)
• Annotation: Synthetic summaries generated by Qwen2.5-7B-Instruct (4-bit quantized)
• Split: 90% train / 10% test

Hyperparameters

• Epochs: 1
• Learning Rate: 2e-4
• Batch Size: 4 (gradient accumulation: 2, effective batch: 8)
• Precision: FP16
• Max Sequence Length: 1024 tokens
• Optimizer: AdamW
• Total Steps: 282

Hardware

• GPU: NVIDIA Tesla T4 (16 GB VRAM)
• Platform: Kaggle Notebooks
• Training Time: ~14 minutes

Evaluation Results

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load model
base_model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "mohammed-el-baraka/KhulasaAI")

# Summarize
text = "your Arabic text here..."
messages = [
    {"role": "system", "content": "لخص النص التالي."},
    {"role": "user", "content": f"النص: {text}\n\nالملخص:"}
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(inputs.to(model.device), max_new_tokens=150, temperature=0.7, do_sample=True)
summary = tokenizer.decode(outputs[0], skip_special_tokens=True).split("assistant")[-1].strip()
print(summary)

Limitations

• Trained on Modern Standard Arabic (MSA) only — performance on dialectal Arabic (Egyptian, Gulf, Moroccan, etc.) is not guaranteed.
• Summaries are limited to ~150 tokens due to max_new_tokens constraint during annotation.
• The model may hallucinate facts not present in the source text, a known limitation of abstractive summarization.
• Input articles were truncated to 2,000 characters during training, so performance on very long documents may degrade.

Citation

@misc{elbaraka2026khulasaai,
  title={KhulasaAI: Arabic Text Summarization via Knowledge Distillation},
  author={Mohammed El Baraka},
  year={2026},
  month={january},
  howpublished={\url{https://github.com/mohammed-el-baraka/KhulasaAI}}
}

Author

Mohammed El Baraka — University Mohammed VI Polytechnic (UM6P)

Framework Versions

• PEFT 0.18.0
• Transformers 4.40+
• PyTorch 2.0+
• TRL 0.8+