barqpulse-v1-qwen2.5-1.5b-lora

BarqPulse — Arabic Comment Intent (LoRA)

A LoRA adapter for Qwen/Qwen2.5-1.5B-Instruct that classifies Arabic customer comments into five intents: Complaint · Question · Suggestion · Praise · Statement.

Trained on real feedback about a Saudi digital payment card. 🔧 Full pipeline and training code

Results

Held-out benchmark of 375 real Arabic comments, labeled by hand.

Read this before using it

Suggestion does not work. F1 is 0.00 — five benchmark examples, so the model never learns the class. If suggestions matter to you, this adapter isn't ready for it. Praise at 0.38 is close to a coin flip.

Training labels came from gpt-4o-mini, not humans. The benchmark was annotated by hand, which is why the failures above are visible rather than hidden — but the ceiling on the working classes is still the teacher's judgment.

Domain: YouTube comments about one fintech product, Saudi and Gulf dialect. Other domains, dialects, and Modern Standard Arabic are untested.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from peft import PeftModel
import torch, json

ADAPTER = "techwithrayoma/barqpulse-v1-qwen2.5-1.5b-lora"

SYSTEM = (
    "You are a strict text classification system.\n"
    "Classify the comment into exactly one of these labels:\n"
    "Question, Complaint, Statement, Praise, Suggestion.\n"
    "Return ONLY valid JSON in this exact format:\n"
    '{"predicted_intent": "<label>"}\n'
    "Do not output anything else."
)

tok  = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", torch_dtype=torch.float16, device_map="auto"
)
pipe = pipeline("text-generation", model=PeftModel.from_pretrained(base, ADAPTER),
                tokenizer=tok, max_new_tokens=32, do_sample=False)

def classify(comment):
    prompt = tok.apply_chat_template(
        [{"role": "system", "content": SYSTEM},
         {"role": "user",   "content": f"Comment:\n{comment}"}],
        tokenize=False, add_generation_prompt=True)
    out = pipe(prompt)[0]["generated_text"][len(prompt):]
    return json.loads(out[out.find("{"):out.rfind("}") + 1])["predicted_intent"]

classify("كم المده بعد الطلب لحتى توصل البطاقه")   # → "Question"

⚠️ The system prompt is not optional. The model was fine-tuned on this exact format. A different one will score well below the numbers above. max_new_tokens must be at least 32 — the JSON needs about 12 tokens, and truncating it breaks every prediction.

Training

Intended use

Triage and analytics on Arabic customer feedback at volume. Not for automated decisions about individual people — two of five classes don't work well enough, and the model hasn't been evaluated for bias across dialects or demographics.

License

Apache 2.0, following the base model.