Downloads · 30 days
131
7% of all-time downloads
Nicolassuez/Qwen3-Reranker-0.6B-W8A16-g128-gptq-resident
Qwen3-Reranker-0.6B-W8A16-g128-gptq-resident is a text ranking model from Nicolassuez. Use it for the text ranking task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
The exact same int8 values and scales as Qwen3-Reranker-0.6B-W8A16-g128-gptq (bit-exact unpack of the GPTQ solution, zero requantization), repackaged as repo custom code (trustremotecode=True) whose Int8GroupLinear mo…
Downloads · 30 days
131
7% of all-time downloads
All-time downloads
1.9K
Public
Parameters
599M
770 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors758 MB · 99%
How the weights are stored.
I8440M · 73%
From the Hugging Face model README
The exact same int8 values and scales as
Qwen3-Reranker-0.6B-W8A16-g128-gptq
(bit-exact unpack of the GPTQ solution, zero requantization), repackaged as repo custom code
(trust_remote_code=True) whose Int8GroupLinear modules keep the int8 weights RESIDENT in VRAM
and dequantize per forward, instead of transformers' decompress-to-bf16-and-keep behavior.
Resident weights: 740 MiB instead of 1136 MiB bf16. lm_head stays bf16 (tied to the embedding
table; the yes/no score is read from it). Fidelity is that of the GPTQ parent: identical bf16 top-1
on every query bf16 itself decides (score gap >= 0.01), 70 queries x 20 candidates.
Load with AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True) and apply the
official Qwen3-Reranker scaffold; score = P("yes") from the last-position logits. Pass
logits_to_keep=1 to avoid materializing full-sequence vocab logits (see the parent model card for
the complete snippet; only the repo id and trust_remote_code=True differ).
Companion embedder: Qwen3-Embedding-0.6B-W8A16-g128-resident.
Qwen3 forward numerics drift between transformers 4.57 and 5.14 (measured on the embedding twin; applies to the bf16 base too). Keep the serving version pinned.