llm-d-sc-genlen-gate

llm-d-sc-genlen-gate

genlen classifier for llm-d semantic classification. Labels: LONG, SHORT.

Architecture: sequence-classification head (requires a runtime that reads logits), base sentence-transformers/all-MiniLM-L6-v2.

Accuracy

Read the real-traffic row first.

Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Both seeds: seed 11 refined gold 0.9569 real gold 0.9532 seed 22 refined gold 0.9542 real gold 0.9506

WHAT THIS MODEL IS. Not a four-tier cost classifier. It answers the binary capacity question the deployed router asks: will this request produce a SHORT or a LONG generation? MINIMAL/LOW fold to SHORT, MODERATE/HIGH to LONG. Tier-exact accuracy on the full cost taxonomy is 0.8976 and is capped near 0.914 by inter-juror agreement; this decision is not.

THIS IS THE BEST-EVIDENCED RESULT IN THE PROJECT, for a reason that is easy to miss: its majority-class baseline is 53.10%. There is no majority shortcut available, so 95.69% is +42.6 points over chance -- unlike the project's other binary gates, whose baselines sit at 83-86%. Per class on refined gold: LONG recall 98.98% / precision 93.30%, SHORT recall 91.95% / precision 98.77%.

IT ALSO REPLICATES ON A SECOND, INDEPENDENT EVAL. On genlen-volume-gold (727 rows, built separately to over-sample high-volume requests) it scores 92.30% against a 59.28% baseline. No other decision measured in this project has a second eval.

WEAKEST POINT, STATED. On the 168 rows where the three-model jury SPLIT, it scores 84.52% against a 55.95% baseline -- still well clear of chance, but 11 points below its clean-subset figure. Jury agreement on those rows is 57.7%.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

427 rows from genlen-v2+genlen-real+genlen-real-contested+genlen-distill, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: None. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 7.94 ms, p99 18.03 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.