qwen3-1.7b-cs-support-lora

Qwen3-1.7B — Customer Support LoRA adapter

A LoRA adapter that fine-tunes Qwen/Qwen3-1.7B to answer customer-support messages (account, orders, refunds, payments, invoices, subscriptions, …) measurably better than the base model. Trained on the Bitext Gen-AI Customer Support dataset.

Served build (merged + quantized): kvn12/qwen3-1.7b-cs-support-gguf.

Results

Held-out 1,100-item test split the model never trained on, served at Q4_K_M via Ollama, judged by Claude Opus 5 (structured rubric). Base is re-baselined through the identical serving pipeline, so only the weights differ.

Improves on every metric across all 11 categories and 27 intents; hallucination drops ~70% relative.

Prompt template (train == serve)

Qwen3 chat format with thinking disabled (enable_thinking=False, i.e. an empty <think></think> block in the assistant turn). Fixed system prompt:

You are a customer support assistant for an online business. A customer has sent you a message. Respond directly to the customer in a helpful, professional, and empathetic tone. Understand what they need, give accurate information, and clearly explain any steps required to resolve their request. Keep the response focused and concise.

Decoding: greedy (temperature=0), seed=42, max_new_tokens=512.

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

SYSTEM = ("You are a customer support assistant for an online business. A customer has "
          "sent you a message. Respond directly to the customer in a helpful, professional, "
          "and empathetic tone. Understand what they need, give accurate information, and "
          "clearly explain any steps required to resolve their request. Keep the response "
          "focused and concise.")

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(base, "kvn12/qwen3-1.7b-cs-support-lora")

msgs = [{"role": "system", "content": SYSTEM},
        {"role": "user", "content": "I was charged twice for order 8842, please help"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Training

LoRA r=16, α=32, dropout=0.05 on q,k,v,o,gate,up,down; 3 epochs (early stopping, patience 2); lr 2e-4 cosine, 3% warmup; effective batch 32; max seq len 1024; completion-only loss (prompt masked). ~497 min on one Kaggle T4 (fp16). Checkpoint chosen by task metrics (judge), not just val loss; val loss 0.60→0.57→0.566.

Limitations

• Reproduces the dataset's {{placeholder}} style (e.g. {{Order Number}}) — de-template or post-process for production.
• Residual "confident fabricated procedure" hallucination (~12%), worst on cancellation-fee queries.
• Metrics are LLM-as-judge (Opus 5) against dataset references, applied identically to base and tuned — fair for comparison, but not human ratings.