Qwen3-Reranker-0.6B-GGUF

Qwen3-Reranker-0.6B — GGUF (NVFP4 / Q6_K)

GGUF quantizations of Qwen/Qwen3-Reranker-0.6B for llama.cpp, built for the llama-server --reranking / /rerank path. These are hybrid quantizations: the 2-way relevance head (cls.output.weight) is kept in BF16, every normalization tensor is kept in F32, and in the NVFP4 build the 155M-row embedding table is kept in Q8_0 — so the yes/no scoring path and the embedding lookup stay at high precision while the bulk of the 0.6B body is compressed. The result is files 2.4–2.9× smaller than the BF16 reference with essentially unchanged ranking scores.

Files

<details> <summary>SHA‑256</summary>

91bbd34601c7b8ddb8a8413cdea90536e9754c010da0be7112246d7441411c7d  Qwen3-Reranker-0.6B.NVFP4.gguf
4e0cb92dd4e1f1c8eb6d9537f95d14b7f89aa48da27d64dc826e5837be2c6b40  Qwen3-Reranker-0.6B.Q6_K.gguf

</details>

Quantization layout

Both files are GGUF v3, architecture qwen3, 311 tensors, 595,778,560 parameters (595.8M). The NVFP4 file is a deliberate per‑tensor mapping, not a vanilla llama-quantize X NVFP4 output:

The design keeps the tensors that most directly shape the final score — the embedding lookup, the 2‑way head, and every norm — at high precision and compresses only the large linear layers. In the NVFP4 build that costs 157.4 MiB for the Q8_0 embed table (vs an all‑NVFP4 build at 325.5 MiB) in exchange for keeping the lookup path out of the NVFP4 mid‑band distortion regime.

Fidelity vs BF16

Q6_K — controlled harness

Measured against a BF16 GGUF reference over 12 queries × 5 passages = 60 reranks (2 relevant, 1 near‑miss, 2 off‑topic each), CPU, 32 threads, flash‑attention on, ctx 1024, no prompt caching. Prompts are built from the identical embedded rerank template, so the comparison isolates quantization error.

Q6 is effectively indistinguishable from BF16 on this workload — deviations are at most a single adjacent swap, and never across the relevant/off‑topic boundary.

NVFP4 (V1) — live memory bank

Measured against the same BF16 reference on a live Hindsight memory bank: 18 queries × 269 stored documents, served through the production llama-server --reranking path.

Reading: on this bank the GPU↔CPU kernel difference (τ ≈ 0.5–0.6) is the dominant term, and the NVFP4‑body deltas sit inside that band — the Q8_0 embed swap (V1 vs V0, same GPU) is of the same magnitude as the device noise floor, so it is not a measurable ranking regression here. The clean quantization statement is the same‑device row: Q8_0 preserves 98.3% of document‑pair orderings vs BF16 (Kendall τ 0.97, median rank shift 3 of 269).

License

Apache‑2.0, same as the source model. Quantized for convenience; the original weights and architecture are © Qwen Team — see Qwen/Qwen3-Reranker-0.6B.

Citation

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}