eva-qwen35-4b-support-merged

Eva Customer Support Manager — merged model

Qwen/Qwen3.5-4B adapted for customer-support triage in a women's health app (cycle tracking, pregnancy, teen mode) operating in Kazakhstan in Russian and Kazakh. A QLoRA adapter was trained separately and merged into the unquantized half-precision base.

Given one customer message, the model emits only a JSON object classifying it on four axes. It does not write prose and is not meant to: in the system it was built for, the JSON selects a reply from an approved template library.

Intended use

Triage and routing. Downstream of this model sits a deterministic safety layer that can only reduce the system's autonomy, never raise it. Do not deploy the model without that layer. It is not a medical device, not a diagnostic tool, and must never answer a medical question on its own.

Output format

{
  "category": "auth_otp",
  "priority": "high",
  "sentiment": "negative",
  "action": "answer_from_kb"
}

The label space was frozen after measuring 214 real support conversations and must not be extended: the baseline and the fine-tuned model are scored against an identical set, or the comparison below is void.

Training

The test split was not used during training.

Training data

Synthetic. The corpus was generated from a catalogue of the app's real defects and a catalogue of behaviours that look like defects but are not — for example an empty "Today" screen after childbirth, or an auto-cancelled unpaid order. That second catalogue is the point: an off-the-shelf model confidently files false bug reports against intended behaviour, and no public corpus contains this knowledge.

Real customer messages were never used for training. They exist only as a held-out slice for measuring behaviour on live traffic.

Evaluation

215 held-out examples. Base model is Qwen/Qwen3.5-4B zero-shot with the same prompt.

Read these numbers with two caveats

1. The base model mostly failed on format, not on understanding: 185 of its 215 answers were invalid JSON. A large share of the category gain is format compliance. Reporting the category number without this line overstates what was learned.
2. The test split is synthetic. A separate slice of 282 hand-labelled real conversations exists and has not been scored yet. Until it is, nothing here is evidence about live traffic.

Class coverage in real data

Four of the fourteen categories have no real examples at all (pregnancy, health_diary) or exactly one (auth_account, age_guardian — the latter safety-critical). Per-class claims about those four are not supportable, and a macro average that hides them would be misleading.

Limitations

• Educational prototype, not a production system and not a medical device.
• Trained on synthetic data; measured on synthetic data.
• Kazakh is under-represented: 18 of 214 real conversations. The system it belongs to never grants Kazakh messages autonomous replies for this reason.
• Deidentification tooling for Kazakh does not exist, and for Russian the only benchmark was machine-translated. Do not assume input is safely scrubbed.
• Prompt injection is bounded by architecture, not by the model: the agent selects from approved templates, so the worst case is a wrong template rather than arbitrary text. Remove the template layer and that bound is gone.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AygerimTan/eva-qwen35-4b-support-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="auto",
)

License

Inherits the base model's licence. Owner: confirm Qwen/Qwen3.5-4B's terms and add the matching license: field to the front matter before treating this card as final. It is deliberately left out rather than guessed.