setu-qwen35-2b-lora

SETU Qwen3.5 LoRA Model Card

Model

SETU fine-tunes a 4-bit MLX conversion of Qwen3.5 for ten-field structured prediction from Hinglish and English e-commerce support messages.

The default learning model is mlx-community/Qwen3.5-2B-MLX-4bit. A single final comparison may use mlx-community/Qwen3.5-4B-MLX-4bit.

The 2B artifact used for the recorded runs resolved to Hugging Face revision 93760be4f1f69842a46bc13dbdc0f19e291392a3.

Status

The 2B reference run completed 800 iterations. Checkpoint 700 was selected using the lowest sampled validation loss and packaged at artifacts/adapters/2b-selected. The 4B config is provided but has not been run; it remains gated on a deliberate capacity experiment.

Selected adapter SHA-256: a390b0e2fa79c81327ca372ea741ace6f7fee141bb23810ec4c4f62daf6bc204.

Task

Input: one Hinglish or English support message.

Output: one schema-valid JSON object containing intent, category, issue type, urgency, sentiment, language mix, three optional entities, and requested resolution.

Training

The base model remains quantized and frozen. LoRA adapters are attached to selected full-attention and linear-attention projections. Prompt tokens are masked from the supervised loss. Gradient checkpointing and accumulation keep the experiment suitable for a 24 GB Apple-silicon machine.

Training data is the synthetic SETU train split. Model selection must use the validation split.

Evaluation

Report:

• Parse and schema validity.
• Exact ten-field object match.
• Mean correct fields.
• Per-field accuracy and macro F1.
• Bootstrap confidence intervals.
• Behavioral slices.
• Inference latency percentiles.

Invalid model output is counted as incorrect. No label-repair layer is used.

The selected adapter achieved 96.33% schema validity and 46.67% exact ten-field match on all 600 held-out synthetic test records. Mean correct fields was 8.99 out of ten, intent accuracy was 94.33%, and issue-type accuracy was 95.50%. See reports/qwen35_2b_selected_test_metrics.json for all fields, confidence intervals, slices, and latency.

Limitations and risks

The training and evaluation messages are synthetic and can contain generator artifacts. The model may fail on code-switching styles, spelling, products, issues, and adversarial inputs outside the generator. Sentiment and language labels can be subjective. The system may copy apparent identifiers and should not receive real personal information in the demo.

SETU must not autonomously execute refunds, account changes, purchases, or other customer actions. A production implementation requires privacy review, authentication, monitoring, fallback behavior, and human escalation.

The largest measured weakness is language-mix classification, especially the Hindi-dominant slice. The synthetic test result must not be generalized to real customer traffic.

Portable serving artifact

The selected MLX adapter was converted to a llama.cpp-compatible F16 LoRA for VAHAAN. Its SHA-256 is 2fc3026f62bb8f1b4ecfb40190d6270870a54bec426833b10ba7fb3e75e20a33. It is served with the Q8_0 artifact from bartowski/Qwen_Qwen3.5-2B-GGUF at revision 7d26695454df6de5fbcce2e58681e62dae06ce43.

Q8_0 was selected over Q4_K_M using a fixed 50-scenario held-out canary. It reached 92% schema validity, 62% exact agreement with MLX predictions, and 8.88 mean matching fields out of ten. The canary measures release equivalence; the 600-row MLX report remains the primary model-quality evaluation.

No human-authored challenge set has been completed. This limitation must remain visible wherever the artifact is published.