llm-d-sc-sensitivity-v3-bge

llm-d-sc-sensitivity-v3-bge

sensitivity classifier for llm-d semantic classification. Labels: PUBLIC, INTERNAL, CONFIDENTIAL, REGULATED, NEVER_EGRESS.

Architecture: sequence-classification head (requires a runtime that reads logits), base BAAI/bge-base-en-v1.5.

Accuracy

Read the real-traffic row first.

Hand-authored minus real traffic: +0.016. Hand-authored held-out prompts are written in the same clean register as the anchors; real users send truncated pastes, fragments and roleplay preambles. The real-traffic row is the one that predicts production behaviour.

Per-tier recall — read this before the accuracy

For an egress or capacity gate the errors are not interchangeable: missing a NEVER_EGRESS prompt leaks a live credential, while over-flagging a PUBLIC one costs a cheap round trip. Aggregate accuracy is carried by the largest class and hides this.

Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Both seeds: seed 11 entsec 0.8204 real gold 0.8908 seed 22 entsec 0.8147 real gold 0.8768 median entsec 0.8176, against 0.7808 for the previously shipped v2 -- +3.68

TWO ONE-LINE CHANGES, MEASURED SEPARATELY THEN STACKED.

1. Tier-escalated class weighting REMOVED. It was introduced as a safety measure and had never been audited against the safety metric that justified it. Swept at 0.0/0.5/1.0 over two seeds the effect is monotone: each step costs about a point of accuracy while buying under a point of gate containment, and it RAISES over-block by 1-2.4 points at every gate.
2. bge-base-en-v1.5 instead of all-MiniLM-L6-v2, worth +4.53 at fixed schedule and corpus. An earlier conclusion that 'bigger is worse' on this signal was measured on ModernBERT and does not hold here.

READ THE GATE NUMBERS, NOT THE ACCURACY. Accuracy ranks this model above MiniLM+esc0, but at MATCHED CONTAINMENT the MiniLM model wins 6 of 9 gate/threshold cells while being 1.70 points worse on accuracy. This has now happened three times in this project: on an ordered taxonomy with a threshold gate, accuracy ranks models differently from the metric the deployment runs on.

USE THIS MODEL for an egress gate run at HIGH containment. At the NEVER_EGRESS boundary it over-blocks 2.39% at 90% containment and 8.19% at 95%, against MiniLM's 3.41% and 9.22%. USE MiniLM+esc0 INSTEAD for REGULATED gating, for low-containment operating points, or when CPU latency matters: this model is 31.2 ms p50 against MiniLM's 8.38 ms, measured in one interleaved process (separately-timed latency figures in this project were contaminated by concurrent load). MIDDLE OPTION: bge-small at the same recipe scores 0.8119 for 13.94 ms.

SCOPE LIMIT. entsec is a SYNTHETIC enterprise eval and is 95.8% distinguishable from real assistant traffic. It exists because real consumer traffic is ~93% PUBLIC and cannot measure the tiers that gate egress. These numbers describe enterprise-LIKE synthetic traffic; there is no real enterprise corpus here to validate them against.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

42237 rows from sensitivity-v2:6783+sensitivity-real:8361+sensitivity-enterprise:31168+sensitivity-real-contested:618, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: {'INTERNAL': 14842, 'REGULATED': 6948, 'PUBLIC': 9831, 'NEVER_EGRESS': 4561, 'CONFIDENTIAL': 6055}. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 23.0 ms, p99 32.44 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.