llm-d-sc-cost

llm-d-sc-cost

A 384-dimensional sentence embedding model fine-tuned to separate expected serving cost, for use as a routing signal by llm-d-sc.

The model does not emit a class directly. It produces an embedding that is ranked against labelled anchors (anchors.json, shipped here), so the taxonomy stays replaceable data rather than a frozen classification head.

Taxonomy

Cost is related to complexity but is not the same signal. Summarising a long document is expensive and simple; a short proof is cheap and hard. This taxonomy tracks the size of the work, not its difficulty.

Intended use

Budgeting and admission decisions: which requests justify a large context window, a long generation budget, or a more expensive endpoint. The classifier emits ranked evidence only; the routing and budgeting decisions remain the caller's.

Evaluation

Evaluated by llm-d-sc on a held-out set of 60 prompts authored independently of the training corpus, 15 per tier, of which 20 are deliberate boundary cases. Method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.

Confusion (rows are truth, columns predicted):

Every error is an ADJACENT-tier confusion. For an ordinal taxonomy that is the benign failure mode: a request misread by one tier gets a slightly wrong budget, not a wrong destination. There are no MINIMAL-to-HIGH style errors.

Latency on CPU (single thread, embed plus rank): p50 8.5 ms, p99 10.1 ms.

Produced on one contributor's homelab; not independently reproduced.

Training

Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and group_by_label batch sampling, 12 epochs, batch size 64, lr 2e-5.

Data: 2650 synthetic prompts generated by Qwen3.8-27B across 40 domains using real-world, boundary and hard-negative strategies, then re-labelled independently by DeepSeek-V4-Flash. Rows where the two models disagreed were dropped, which removed 34% of the corpus, leaving 1416 balanced examples (354 per tier). A 34% disagreement rate is itself informative: cost tiers are genuinely more ambiguous than complexity tiers, and the dropped rows were mostly MODERATE/HIGH boundary cases.

The held-out evaluation set was authored by hand before the training data existed and is asserted to share no prompt with it.

Limitations

• English only.
• Trained on synthetic data; no human-labelled validation set exists.
• Cost depends on your serving stack. These tiers describe the size of the requested work, not the price of any particular deployment.
• Anchor quality directly determines accuracy. Replacing anchors.json changes behaviour without retraining.

License

Apache-2.0.