Qwen3-Embedding-0.6B-Q8-NVFP4

Qwen3-Embedding-0.6B-Q8-NVFP4

Qwen3-Embedding-0.6B · NVFP4 quantized · 507 MB

A hand-built GGUF of the Qwen3-Embedding-0.6B text embedding model, quantized with a mixed NVFP4 recipe for Blackwell GPUs. 1024-dim embeddings, 32K context, last-token pooling.

Credits

• The Qwen team (Qwen/Qwen3-Embedding-0.6B) — built the base model.
• AIconjured — the NVFP4 quantization and packaging.

Files

Size: 507 MB — down from 1198 MB (F16) and 639 MB (Q8_0).

Quantization recipe

Mixed-precision, built with llama.cpp's --tensor-type-file plus an imatrix generated from ~144 KB of mixed calibration text:

Recipe (llama.cpp --tensor-type-file format):

token_embd\.weight=q8_0
output_norm\.weight=q8_0
blk\.\d+\.ffn_gate\.weight=nvfp4
blk\.\d+\.ffn_up\.weight=nvfp4
blk\.\d+\.ffn_down\.weight=nvfp4

Measured quality

Embedding cosine similarity vs the F16 original, measured on 20 diverse texts through the actual llama.cpp runtime (last-token pooling, 1024-dim):

Short factual queries stay tight (~0.98); harder semantic queries drift up to ~18°. In retrieval, expect a small recall drop concentrated on nuanced queries. If you need near-lossless embeddings, use the Q8_0 version instead.

Usage

Ollama:

FROM ./Qwen3-Embedding-0.6B-NVFP4.gguf

llama.cpp:

llama-server -m Qwen3-Embedding-0.6B-NVFP4.gguf --embedding -c 32768
# POST /embeddings {"input": "your text"}

Hardware notes

• Built and verified on 2× RTX 5060 Ti (Blackwell, sm_120a) with a CUDA 12.8 llama.cpp build.
• NVFP4 kernels use the native Blackwell FP4 path; on non-Blackwell GPUs the tensors still load and run (dequantized), with less speed advantage.
• The model is tiny (507 MB) — fits entirely in a single GPU with large context headroom.

Reproduction

• llama.cpp: 378aa2ebc (2026-08-27), CUDA build
• Quantize: llama-quantize --imatrix imatrix-emb.gguf --tensor-type-file recipe-emb-nvfp4.txt ...
• Imatrix: 32 chunks × 2048 tokens, mixed wiki/prose calibration

License

Apache-2.0, matching the base Qwen/Qwen3-Embedding-0.6B model.