intent-router-zh-setfit-v1

intent-router-zh-setfit-v1

SetFit intent router for AI coding-agent traffic, fine-tuned from Qwen/Qwen3-Embedding-0.6B. It maps a user prompt to 21 intents across coding / ops / general_control plus out_of_scope, and ships with a v4 routing policy (temperature scaling + optional top-2 flips + precision-first keyword overrides).

基于 Qwen3-Embedding-0.6B 的中英双语意图路由器（21 类）。
主评测集 gold_test n=695；独立短指令 coding OOD n=360；action–object probe n=45。
发布数字（CPU 复算，与 metrics.json 一致）：closed-set 0.705 / routed v4 0.675 / domain 0.847。

Datasets & test sizes (release record)

Gold test composition (n=695)

Length (characters): min 4 · median 290 · mean 1418 · max 36248.

Per-class support on gold (closed-set evaluation order):

Independent coding OOD (n=360)

Action–object probe (n=45)

Held-out triples that keep the object fixed and vary the action intent (e.g. explain / search / monitor the same resource). Accuracy 0.978, macro-F1 0.741.

Release status

Labels

Training data (original train split, n=3203)

Training recipe oversamples the head to min 150/class (n_head_train=4437) and uses a capped class-balanced contrastive body set (n_body_train=3640). Oversampling cannot invent evidence for long-tail classes.

Intended use

• Route prompts of an AI coding assistant to specialized handlers.
• Best on long, context-rich, agent-style prompts (system context, diffs, stack traces). On gold prompts ≥800 chars, closed-set accuracy is 0.782.
• Max input length for best quality: 1024 tokens (training setting).
• Not a general short-prompt router without keyword rules / OOS fallback.

Usage (SetFit / PyTorch)

import os
os.environ["SETFIT_DEVICE"] = "cpu"
import torch
from setfit import SetFitModel

model = SetFitModel.from_pretrained(
    "snival/intent-router-zh-setfit-v1",
    trust_remote_code=True,
)
model.to(torch.device("cpu"))

body = model.model_body
body.max_seq_length = 1024
body.tokenizer.model_max_length = 1024
transformer = body[0]
transformer.max_seq_length = 1024
transformer.tokenizer.model_max_length = 1024
transformer.tokenizer.padding_side = "left"

probas = model.predict_proba([
    "帮我 review 一下这个 PR，重点关注线程安全",
    "Traceback (most recent call last): ... NullPointerException at OrderService.java:148",
])

Framework-free path: encode with the body, then apply head_coef.npy / head_intercept.npy + softmax.

ONNX / ONNX Runtime (CPU)

Encoder graphs are shipped for edge/CPU deployment. The head is not fused into ONNX — apply head_coef.npy + head_intercept.npy after sentence_embedding. Details: ONNX_INFERENCE.md.

I/O: input_ids / attention_mask (int64, [batch, sequence]) → sentence_embedding (float32, [batch, 1024]); pooling lasttoken, L2 normalize True, left padding, max_length=1024.

import json
from pathlib import Path

import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

model_dir = Path(
    snapshot_download(
        "snival/intent-router-zh-setfit-v1",
        allow_patterns=[
            "encoder-int8.onnx",
            "head_coef.npy",
            "head_intercept.npy",
            "head_meta.json",
            "tokenizer*",
            "vocab.json",
            "merges.txt",
            "added_tokens.json",
            "special_tokens_map.json",
            "chat_template.jinja",
            "config.json",
        ],
    )
)

tok = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
tok.padding_side = "left"
sess = ort.InferenceSession(
    str(model_dir / "encoder-int8.onnx"),
    providers=["CPUExecutionProvider"],
)
coef = np.load(model_dir / "head_coef.npy")
intercept = np.load(model_dir / "head_intercept.npy")
labels = json.loads((model_dir / "head_meta.json").read_text(encoding="utf-8"))["labels"]

enc = tok(
    ["帮我 review 一下这个 PR，重点关注线程安全"],
    padding=True,
    truncation=True,
    max_length=1024,
    return_tensors="np",
)
emb = sess.run(
    ["sentence_embedding"],
    {
        "input_ids": enc["input_ids"].astype("int64"),
        "attention_mask": enc["attention_mask"].astype("int64"),
    },
)[0]
logits = emb @ coef.T + intercept
probs = np.exp(logits - logits.max(axis=-1, keepdims=True))
probs /= probs.sum(axis=-1, keepdims=True)
print(labels[int(probs.argmax())], float(probs.max()))

End-to-end routing (recommended)

Ship files: labels.json, metrics.json, rejection_policy.json (v4).

1. Keyword hard-routing (657 rules in this release)
2. Optional top-2 flip rules for known near-neighbors
3. Temperature scaling (temperature ≈ 1.745)
4. Short-text / model-OOS → out_of_scope (short_text_max_chars=3; score-threshold rejection is off: max_probability=0.0)

Reference: routing_policy.decide_route(). On gold n=695, route reasons were:

Smoke examples (CPU)

Evaluation

All primary numbers below were recomputed on CPU from the shipped model + eval_proba_695_Qwen--Qwen3-Embedding-0.6B.npy and match metrics.json.

1) Gold test — n=695 (primary)

By length (chars):

Intent accuracy by true domain:

Per-class closed-set (precision / recall / F1 / support):

Top closed-set confusions:

Action–object probe (n=45): accuracy 0.978, macro-F1 0.741.

Calibration (policy metadata): after temperature scaling, ECE ≈ 0.055, NLL ≈ 1.103 (before: ECE ≈ 0.192 / NLL ≈ 1.466).

Prefer the 695-row gold set + metrics.json as the release record. The older 611-row split is not identical (overlap 518 texts).

2) Independent coding OOD — n=360 (short-instruction stress)

Zero leakage vs train/gold. Median length 204 chars (short-heavy). Source report: coding_test_report.md / coding_eval_results.json.

Per-class routed F1 (best → worst): code_review 0.548 · code_repair 0.333 · code_search 0.256 · code_modification 0.254 · code_authoring 0.222 · test_generation 0.222 · architecture_consultation 0.188 · code_explanation 0.187 · dependency_management 0.000.

Routed gain is mostly keyword rules; a large share is honestly rejected or corrected rather than pure argmax. This set is the model’s weakest region.

3) CPU speed (Intel i9-14900KF, 32 threads; coding eval)

Full cold from_pretrained on this machine is ~tens of seconds depending on disk/cache.

Limitations

• Short standalone prompts are OOD. Training is dominated by long agent-style prompts. Short imperatives collapse in embedding space; do not deploy as a general short-prompt router without short-instruction training data.
• Long-tail intents remain weak: dependency_management, test_generation, pipeline_operation (and other low-support classes).
• Cross-domain near-neighbors: code_repair ↔ log_analysis, code_modification ↔ config_change, code_authoring ↔ code_modification, code_search ↔ code_explanation.
• Keyword overrides help explicit patterns but can override a higher-probability neighbor; audit mined rules before production.
• Score-threshold rejection is disabled (max_probability=0.0); re-tune if you need aggressive abstention. OOS support on gold is only 7 rows.
• On gold, v4 routed accuracy is slightly below closed-set (−3.0pp): keywords fix specific phrases more than they lift overall accuracy.

Training

• Method: SetFit (contrastive body + linear SetFitHead)
• Base: Qwen/Qwen3-Embedding-0.6B, max_length=1024, pooling=lasttoken, padding_side=left
• Data: train 3203 → body 3640 (capped) / head 4437 (min 150/class); calibration 473; rule validation 416
• Head: lr=5e-3, l2=0.01, 8 epochs, head_class_weighting=none; body 2 epochs, 8 iterations; batch sizes [2, 64]; AMP + gradient checkpointing
• Device: CUDA (NVIDIA GeForce RTX 5090 D); inference reference target is CPU
• Keyword mining: 647 accepted at train time → 657 rules in shipped rejection_policy.json

Files

Not uploaded by default (local eval assets): gold_test.jsonl (n=695), eval_proba_695_*.npy, train logs, conflict dumps.

Comparison vs first trained release (v1)

Baseline: post-training report on gold n=611 (2026-08-09).
Current: gold n=695 (metrics.json). Write-up: docs/v1_to_current_comparison.md.

Length closed-set: <100 +22.8pp · 100–800 +25.8pp · 800+ +4.9pp.

On 518 overlapping texts present in both gold sets, current closed-set is 0.710 (≈ +15.2pp vs v1 closed 0.558) — main gain is classifier quality, not only gold composition. Independent short-instruction coding OOD (n=360) is unchanged (closed 0.117 / routed 0.203).

How this card was verified

• Recomputed closed-set + v4 routed metrics on gold n=695 from the shipped probability cache; matches metrics.json (closed 0.7050, routed 0.6748, domain 0.8475).
• Gold label / length / domain counts taken from intent-router/gold_test.jsonl (n=695).
• Coding OOD numbers from coding_eval_results.json / coding_test_report.md (n=360; prior CPU↔CUDA argmax agreement 100% on gold).
• Action–object probe from action_object_probe_metrics.json (n=45).
• ONNX artifacts documented in ONNX_INFERENCE.md (fp32 / int8 / woq8).