llm-d-sc-reasoning-gate

llm-d-sc-reasoning-gate

reasoning classifier for llm-d semantic classification. Labels: NO, YES.

Architecture: sequence-classification head (requires a runtime that reads logits), base sentence-transformers/all-MiniLM-L6-v2.

Accuracy

Read the real-traffic row first.

Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Rule declared before uploading, because this project has already had to correct a published figure that was best-of-seeds. Both seeds:

seed 11 refined gold 0.9654 real gold 0.9593 YES recall 0.9815 seed 22 refined gold 0.9707 real gold 0.9665 YES recall 0.9815

WHAT THIS MODEL IS. It is NOT a complexity tier classifier. It answers one binary question that the deployed llm-d-sc router actually asks: does this request need a reasoning model? The four-tier complexity taxonomy is finer than the branch the router takes -- SIMPLE/MEDIUM and COMPLEX/REASONING map to the same backends -- so a tier-exact score charges the classifier for distinctions the deployment discards. Tier-exact accuracy on the full taxonomy is 0.8963 and is capped near 0.926 by inter-juror agreement; this decision is not.

READ THE BASELINE. YES is 3.2% of the training corpus and about 15% of the eval, so the majority-class baseline on refined gold is 85.64%. 96.54% clears it by 10.9 points, and YES recall is 98.15% -- the minority class is where the work is happening, not a rounding artifact.

WEAKEST POINT, STATED. On the 176 rows where the three-model jury SPLIT, this model scores 89.77% against a 90.91% majority baseline -- i.e. below baseline. It fires on all 16 true YES cases there but at 47% precision. Contested rows are about 30% of real traffic for this signal, so that is a real limitation.

RUN THIS AT THRESHOLD 0.75, NOT ARGMAX. Choosing the operating point by 2-fold cross-validation over 40 random splits -- threshold picked on one half, accuracy reported on the other -- gives 97.89% against 96.54% at argmax, +1.35 points for one config value and no retraining. The selected threshold clusters hard at 0.75 (56 of 80 folds), which is what distinguishes this from the three sibling gates whose tuned 'gains' were +0.15, +0.03 and -0.00 with thresholds scattered across the grid. At 0.75 the model keeps YES recall at 98.15% while cutting false fires from 3.73% to 2.80% -- it improves on both axes rather than trading them.

The fitted optimum on the full eval is 98.14% at threshold 0.8. That number is NOT quoted as a result: it was chosen on the rows it is scored on. 97.89% is the cross-validated estimate and the difference is the selection bias.

Precision at argmax on refined gold is 81.5%: it over-routes 3.7% of non-reasoning prompts to the expensive path. For this decision that is a cost trade rather than a correctness failure, and the threshold can be moved if the trade is wrong for your deployment.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

418 rows from reasoning-v2+reasoning-real+reasoning-active+reasoning-distill+reasoning-real-contested, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: None. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 6.7 ms, p99 18.7 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.