northwind-support-lora

Northwind Support LoRA (Qwen3-4B)

A LoRA adapter that moves a 992-token customer-support system prompt — a 20-entry policy handbook plus a strict JSON output contract — into the weights, so inference runs on a one-line system prompt instead.

Trained on a synthetic dataset generated from scratch by a seeded, rule-based generator — no teacher model, no API key, reproducible byte-for-byte. Everything needed to judge the result is on this page: the four-arm comparison, the training setup, and the limitations.

⚠️ This adapter encodes the policies of Northwind Outdoors, a fictional retailer. It is a methodology demonstration, not a usable support agent for any real company. See Limitations.

What it does

Given a customer message, it replies in 2-4 policy-grounded sentences and appends exactly one machine-readable ticket:

Clearance and final-sale items are sold as-is, so the Ridgeline rain shell on NW-771904 can't be
returned or exchanged. I know that's not the answer you wanted; it's called out at checkout.

```json
{"intent": "return_refund", "priority": "P3", "escalate": false, "policy_ids": ["RET-02"], "next_action": "answer_only"}
```

Results

200 held-out conversations, greedy decoding, rule-based scoring against gold labels derived by rule (no LLM judge). Customer phrasings in the test set never appear in training.

91% fewer prompt tokens on every request, while beating the prompted baseline on every accuracy metric. The base + one-line prompt column is the ablation: without the adapter, the base model has no idea the policy IDs or the ticket format exist.

The most useful gap is escalation recall. Both prompted arms are precise (~90%) but miss ~43% of cases that require a human — injury, fraud, chargebacks, legal threats. The adapter reaches 87.5%.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "blackburn1910/northwind-support-lora")

messages = [
    {"role": "system", "content": "You are the customer support assistant for Northwind Outdoors."},
    {"role": "user", "content": "I want to return the rain shell I bought on clearance, order NW-771904."},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False).to(model.device),
                     max_new_tokens=300, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The system prompt must be exactly You are the customer support assistant for Northwind Outdoors. — that is what it was trained against. Qwen3 must run in non-thinking mode (enable_thinking=False).

Training

Training data is generated offline by a seeded, rule-based generator — no teacher model and no API key — so it reproduces byte-for-byte.

Limitations

• It has learned one fictional company's handbook, not customer support. Point it at a different policy set and it will be confidently wrong, in the way a prompted model would not be — the prompted baseline simply follows whatever prompt you give it. That trade is the whole point of the experiment, and it cuts both ways.
• 48% of its replies reproduce a training answer verbatim (after normalising order IDs and numbers), against 0% for the prompted baseline. The training targets are template-generated, so the adapter has substantially memorised the response bank. Read the accuracy figures as evidence about format and policy fidelity, not about writing quality.
• Prose quality is not evaluated. No human raters, no judge model — deliberately, so every reported number is rule-checkable.
• Test phrasings are unseen but come from the same generator as training. This measures generalisation across wording, not across distribution. Real ticket text has a tail of weirdness this data does not.
• Single seed, single base model. Differences of a point or two are noise.
• Not safety-tuned for production use. The escalation rules here are deliberately crisp; real escalation policy is fuzzier and would need human-labelled data.

License

Apache-2.0, matching the Qwen3-4B base model.