Downloads · 30 days
3
1% of all-time downloads
Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident
Qwen3-Embedding-0.6B-W8A16-g128-resident is a feature extraction model from Nicolassuez. Use it when you need embeddings to search or compare text. The card lists the license as apache-2.0.
Same int8 weight-only groupsize-128 quantization as Qwen3-Embedding-0.6B-W8A16-g128, but packaged as repo custom code (trustremotecode=True) whose Int8GroupLinear modules keep the int8 weights RESIDENT in VRAM and deq…
Downloads · 30 days
3
1% of all-time downloads
All-time downloads
297
Public
Parameters
599M
1.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors603 MB · 98%
How the weights are stored.
I8596M · 99%
From the Hugging Face model README
Same int8 weight-only group_size-128 quantization as
Qwen3-Embedding-0.6B-W8A16-g128,
but packaged as repo custom code (trust_remote_code=True) whose Int8GroupLinear modules keep
the int8 weights RESIDENT in VRAM and dequantize one layer at a time inside forward.
Why: transformers decompresses compressed-tensors checkpoints to bf16 in VRAM (lazily, at first
forward) and keeps the bf16, so the stock format saves download but not resident memory. This
packaging saves both.
Measured (H100):
| bf16 | this repo | |
|---|---|---|
| Resident weights (transformers, both after forwards) | 1136 MiB | 740 MiB |
Infinity 0.0.77, --batch-size 8, process VRAM | 2490 MiB | 2102 MiB (-16%) |
| Latency in-process, batch 1 / batch 32 | 17.2 / 23.3 ms | 23.0 / 29.8 ms |
Fidelity: quantization identical to the W8A16-g128 repo (recall@10 0.976 vs bf16 on a 360-doc multi-script index; embedding cosine 0.999 vs the bf16 service).
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident",
trust_remote_code=True)
or AutoModel.from_pretrained(..., trust_remote_code=True) with last-token pooling + L2 normalize.
Works in Infinity:
infinity_emb v2 --model-id Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident \
--trust-remote-code --no-bettertransformer --device cuda
(--no-bettertransformer is required: Infinity 0.0.77's BetterTransformer probe crashes when
optimum is absent or >= 2.0.)
nn.Module.to(dtype) only casts floating-point tensors, so the int8 buffers survive the
.to(float16/bfloat16) calls sentence-transformers and Infinity make.
Qwen3 forward numerics drift between transformers 4.57 and 5.14 (embedding cosine ~0.973 between the two, measured on this model; the drift exists for the bf16 base too). Embeddings indexed with one version should be queried with the same version.
The embedding table (151669 x 1024, 26% of parameters) is now ALSO stored int8, with one bf16 scale per row (a lookup gathers single rows, no accumulation crosses the quantized axis, so per-row scaling is the natural granularity). Resident weights drop 740 -> 592 MiB. Validated on the 360-doc / 70-query benchmark against bf16: mean weight cosine unchanged (0.99922 vs 0.99923 with the bf16 table) and every top-1 change happens on candidate pairs the bf16 model itself separates by less than 0.01. Do NOT replicate this on the reranker: its lm_head is tied to the table and the yes/no score reads it directly.