intent-router-zh-setfit-v2

intent-router-zh-setfit-v2

SetFit intent router for AI coding-agent traffic. Fine-tuned from jinaai/jina-embeddings-v5-text-small-classification (Qwen3-0.6B backbone, last-token pooling, task prompts). It maps a user prompt to 21 intents across coding / ops / general_control plus out_of_scope, and ships with a v5 routing policy (temperature scaling, argmax routing — see Routing policy).

License note: this checkpoint is fine-tuned from a CC BY-NC 4.0 base model, so the derived weights are non-commercial only.

主评测集 gold_test n=695： closed-set / routed 0.699 · macro-F1 0.554 · domain 0.875。
独立短指令 coding OOD n=360：closed 0.783。
弱类长尾切片 n=180：closed 0.694。
action–object probe n=45：1.000。

Datasets & test sizes (release record)

Synthetic additions come from the LLM augmentation program (planned 936: 216 short coding instructions, 580 weak-class workflows, 140 near-neighbor boundary cases) plus long-tail, minority, OOS-hard-negative, and short Chinese code-search generators. Reserved sets (gold, calibration, probes, long-tail slice) were excluded from generation by contract.

Train distribution (n=5237)

Gold composition: coding 466 (67.1%) / ops 166 (23.9%) / general_control 56 (8.1%) / out_of_scope 7 (1.0%); length median 290 chars, max 36248.

Labels

Intended use

• Route prompts of an AI coding assistant to specialized handlers (zh / en / mixed, short imperatives through long agent-style context).
• Max input length for best quality: 1024 tokens (training setting).
• Not a general-purpose NLU router: OOS support in gold is only 7 rows, and general_control remains the weakest domain (0.500 intent accuracy on gold).

Usage (SetFit / PyTorch)

import os
os.environ["SETFIT_DEVICE"] = "cpu"
import torch
from setfit import SetFitModel

model = SetFitModel.from_pretrained(
    "snival/intent-router-zh-setfit-v2",
    trust_remote_code=True,
)
model.to(torch.device("cpu"))

body = model.model_body
body.max_seq_length = 1024
body.tokenizer.model_max_length = 1024
transformer = body[0]
transformer.max_seq_length = 1024
transformer.tokenizer.model_max_length = 1024
transformer.tokenizer.padding_side = "left"

probas = model.predict_proba([
    "帮我 review 一下这个 PR，重点关注线程安全",
    "cargo add serde --features derive，然后更新 lockfile",
])

Framework-free path: encode with the body, then apply head_coef.npy / head_intercept.npy + softmax. For CPU/edge deployment without PyTorch, see ONNX / ONNX Runtime.

ONNX / ONNX Runtime (CPU)

Use the encoder graphs shipped with this checkpoint; ONNX exports from other checkpoints are not compatible. Prompt contract: the Jina v5 base prepends the prompt prefix "Document: " to every text inside SentenceTransformer.encode() — prepend the same prefix before tokenizing (plain string concatenation) or embeddings drift from the SetFit-native path. The prefix is recorded in head_meta.json (prompt_prefix). Left padding, max_length=1024; the head is not fused into ONNX — apply head_coef.npy + head_intercept.npy + softmax after sentence_embedding. Details: ONNX_INFERENCE.md.

Not shipped: dynamic-int8 (quantize_dynamic) fails on this base model — quantizing the token-embedding table (single-scale UINT8) or per-channel MatMul weights distorts embeddings (gold argmax agreement 92.5–92.9%, cosine p5 ≈ 0.8). quantize_int8.py now enforces a real-data release gate and refuses such artifacts; use quantize_woq8.py output instead.

Speed on the eval CPU (i9-14900KF, 32 threads; ONNX Runtime CPUExecutionProvider):

On this CPU, woq8 trades ~23% throughput for a 3.9× smaller artifact; pick fp32 for latency-critical serving and woq8 where footprint dominates.

End-to-end routing (recommended)

Ship files: labels.json, metrics.json, rejection_policy.json (v5). Reference: routing_policy.decide_route().

1. Temperature scaling (temperature ≈ 2.396)
2. Short-text rejection is disabled: it also requires a positive score threshold, while max_probability=0.0
3. Score-threshold rejection is off (max_probability=0.0, min_margin=0.0)
4. Keyword and flip routing are disabled (enable_keyword_rules=false, enable_flip_rules=false)

The shipped v5 policy is therefore calibrated argmax routing. Do not claim an active OOS/short-text safety fallback unless thresholds are re-calibrated and enabled on production traffic.

Why disabled: on gold, enabling keyword rules drops routed accuracy from 0.6993 to 0.6705 — the base model makes them net-negative. The mined patterns are also near-verbatim phrases from the training corpus, so they are not shipped in rejection_policy.json (keyword_rules: []).

Evaluation

All numbers below were recomputed on CPU (Intel i9-14900KF, 32 threads, PyTorch fp32, max_length=1024) from the shipped checkpoint and cached probability arrays. CPU↔training-machine agreement: closed-set 0.6993 vs metrics.json 0.6964 (2/695 samples differ — numeric drift only).

1) Gold test — n=695 (primary)

Routed equals closed-set because v5 is argmax-only (no keyword/flip overrides); on gold, route reasons were 691 argmax + 4 predicted_out_of_scope.

By length (chars):

Intent accuracy by true domain (closed = routed):

Per-class routed (precision / recall / F1 / support):

Top closed-set confusions: code_search → code_explanation (14) · code_authoring → code_modification (12) · code_repair → log_analysis (7) · code_explanation → code_modification / → code_review (7) · config_change → code_modification (7).

Calibration: raw ECE 0.276 / NLL 2.534 → after policy temperature (T=2.396) ECE 0.102 / NLL 1.334. The model is strongly overconfident: 65.6% of gold errors carry confidence ≥ 0.99. Treat raw scores as rankings, not probabilities; re-fit temperature per deployment if you need abstention.

2) Long-tail coverage slice — n=180 (weak-class stress)

Held-out rows for the 9 historically weakest classes (20 each), never in train.

Per-class recall: infrastructure_provisioning 0.85 · workflow_control 0.85 · incident_response 0.80 · pipeline_operation 0.70 · security_operation 0.70 · test_generation 0.65 · dependency_management 0.60 · general_inquiry 0.35 (top confusions: general_inquiry → architecture_consultation 7, dependency_management → code_modification 4).

3) Independent coding OOD — n=360 (short-instruction stress)

A frozen independent set of short standalone coding instructions (median 204 chars; leakage re-verified 0 against train/calibration/ rule-validation/gold/probe).

Slices: easy 0.782 / medium 0.806 / hard 0.722; zh 0.775 / en 0.791 / mixed 0.781. Dominant confusion: dependency_management → architecture_consultation (17).

4) Action–object probe — n=45

Held-out triples holding the object fixed while varying the action (explain / search / monitor the same resource): accuracy 1.000, macro-F1 1.000.

5) CPU speed (Intel i9-14900KF, 32 threads, fp32)

Expect ~1.2 s/batch at the 1024-token ceiling. ONNX CPU paths (fp32 and woq8) are benchmarked in the ONNX section.

Limitations

• Overconfident: calibrated ECE 0.102 with 65.6% of errors at ≥0.99 confidence. Do not use raw max-probability as an abstention threshold without re-calibration.
• general_control is the weakest domain (gold intent accuracy 0.500); general_inquiry recall is 0.21–0.35 across sets.
• pipeline_operation is 0.000 F1 on gold (support 6) despite 0.70 recall on the long-tail slice — gold samples are agent-style, the slice is imperative-style.
• code_search recall on gold is 0.375 (top confusion → code_explanation).
• dependency_management recall collapses to 0.35 on architecture-flavored phrasing (17/40 → architecture_consultation on the coding OOD set).
• OOS support on gold is only 7 rows; OOS recall 0.286. Score-threshold rejection is disabled.
• CC BY-NC 4.0 (inherited from the Jina v5 base): non-commercial use only.
• Local transformers may warn about a Mistral tokenizer regex when loading this tokenizer; measured effect on this eval was ≤2/695 samples.

Training

• Method: SetFit (contrastive body + linear SetFitHead)
• Base: jinaai/jina-embeddings-v5-text-small-classification (do not substitute the generic jinaai/jina-embeddings-v5-text-small repo; its custom_st wrapper runs under torch.no_grad())
• Contract: max_length=1024, pooling=lasttoken, L2-normalize, padding_side=left
• Data: pool 7284 (5767 real + 1517 synthetic, min 150/class) → train 5237; body contrastive set capped at 250/label (4306 rows); head trained on the full 5237 split with balanced class weights
• Head: lr=5e-3, l2=0.01, 8 epochs; body: 2 epochs, 8 iterations, batch [2,64], AMP + gradient checkpointing
• Device: CUDA (NVIDIA GeForce RTX 5090); inference reference target is CPU
• Keyword mining: 62334 candidates → 20343 rejected for cross-label conflict, 41105 for low support → 740 accepted; disabled in the v5 policy after validation showed a −2.9pp routed drop, and stripped from the shipped policy file (training-corpus fingerprints)

Files

Not uploaded by default (local-only assets): training scripts, training and evaluation datasets, gold_test.jsonl, eval_proba_*.npy, train logs, conflict dumps, and dataset audits. The uploader uses a strict inference-only allowlist and verifies release_manifest.json before upload.

How this card was verified

• Closed-set + v5 routed metrics recomputed on CPU from the shipped checkpoint (evaluate_model.py, cached eval_proba_695_*.npy); matches metrics.json within 2/695 samples.
• Long-tail slice via EVAL_SET_PATH=dataset_splits/eval_long_tail.jsonl; coding OOD via run_coding_eval.py on the frozen test_coding.jsonl (leakage re-verified 0 against all release splits).
• Keyword-rule ablation (kw on/off) recomputed from cached probabilities; policy decisions from routing_policy.decide_route with the shipped rejection_policy.json.
• Calibration (raw vs T=2.396) computed from the same cached probabilities.
• Dataset counts from dataset_splits/release_695/manifest.json, intent-router/dataset_audit.json, and intent-router/metrics.json.
• The public release is generated with python build_release_manifest.py --include-onnx and the uploader verifies every selected inference artifact against that manifest. Training code and datasets are outside the uploader allowlist.

Demo URL: https://intent-api.sxueck.com/demo