SKILLRET-Edge-22M-drift-mixed

SKILLRET-Edge-22M-drift-mixed

Engineering blog: Skill Retrieval on a CPU: Building the SKILLRET Small Model Family (한국어: CPU에서 도는 스킬 검색)

The 22.7M-parameter SKILLRET-Edge bi-encoder for agent skill retrieval, quantized to a mixed-precision layout chosen by a label-free drift signal instead of one uniform bit width. Same architecture, same tokenizer, same query contract as ThakiCloud/SKILLRET-Edge-22M-int4 — this repo exists to show a different allocation technique, not to replace the int4/int3 variants.

Scored on the public ThakiCloud/SKILLRET test split (4,392 queries / 6,006 skills).

NDCG@10 74.57 · 14.1 MB packed (the int3 file size, 99.1% of fp16)

Where it sits

Paired query-level bootstrap on the same 4,392 queries (5,000 resamples), all arms built and scored under one protocol (int3 uniform comes from the 5.2-bit run of the same series):

Read the first row honestly: the gain over uniform int3 is real but under one point. The value of the technique is the ordering (random allocations at the same bytes lose 8–9 points), and the budgets uniform cannot express — at 4.5 real bits (~12.7 MB) the drift allocation keeps 70.42 where uniform int2 gives 59.73 and uniform int3 does not fit.

How the bits were allocated

Each of 12 allocation units was quantized alone at 2/3/4 bits while the rest stayed fp32, and the resulting drift of the output embeddings — mean (1 − cosine) between full-precision and quantized embeddings, averaged over 1,500 calibration skills and 150 calibration queries, balanced across the two — was recorded. No relevance labels are used. An exhaustive integer program then picks the per-unit bit width that minimises predicted drift under a hard byte budget (fp16 scale + zero per group of 16 counted). The drift ordering reproduced across disjoint calibration halves at every size tried (Spearman 1.000 at 75/150/300/600 queries); the label-based ordering on this model did not (−0.29 to +0.34), which is why the label-supervised row above is reported but not recommended.

The picture is the one you would guess after seeing the sensitivities: the word-embedding table is 51.6% of the parameters and its lower-frequency rows barely move the output at 2 bits, while the middle attention/FFN blocks are where 2-bit damage concentrates.

Files

The safetensors path runs fp32 arithmetic; latency is identical to the int4/int3 cards (same architecture, same dequantized format) and is not an integer-kernel speed.

Usage

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("ThakiCloud/SKILLRET-Edge-22M-drift-mixed")
q = m.encode(["I need to convert a spreadsheet into a chart"], normalize_embeddings=True)
d = m.encode(["Chart Builder — turn tabular data into bar/line charts"], normalize_embeddings=True)
print(q @ d.T)

Caveats

• Standard error on this split is about ±0.45; the drift-vs-int3 gap is significant only because it is paired on the same queries. Do not read it as a ranking between cards scored in different runs.
• The protocol matches the other Edge cards: CLS pooling, L2-normalised, max_length=256, bare queries.
• Mixed precision means a real integer kernel has to handle per-tensor (and, for the embedding table, per-row-band) bit widths. That is the cost you pay for +0.65pp at int3 bytes.
• The base student, the distillation recipe and the int4/int3 measurements are documented on ThakiCloud/SKILLRET-Edge-22M-int4.

License

Apache-2.0, inherited from the base model.

Citation

Benchmark: SkillRet, arXiv:2605.05726

Student and uniform-PTQ measurements: Where Post-Training Quantization Breaks Text Embedders, arXiv:2609.16391

Allocation signal (the drift criterion and the allocator used here): Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders, arXiv:2610.09227

⚠️ Read the +0.65pp above next to that paper's held-out result. Configuration-level drift orders sampled mixed-precision plans against held-out retrieval quality at a macro Spearman of 0.911, and the allocation avoids the collapses a transferred signed-geometry criterion produces. But on three embedders sealed until the method, baselines and hypotheses were frozen, drift at the main budget landed below same-budget uniform precision on all three (-0.99, -0.85, -1.01 points). Output drift is a robust coarse sensitivity signal, not a universally optimal allocation objective: the gain reported on this card is measured on this 22M student and this split, and a strong uniform operating point is still the thing to beat.

@article{han2026quantizebydrift,
  title   = {Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders},
  author  = {Han, Hyojung and Kim, Jongmin and Jeon, Seung-Hun},
  journal = {arXiv preprint arXiv:2610.09227},
  year    = {2026}
}