Qwen3-Embedding-4B-F16-GGUF

Qwen3-Embedding-4B — F16 GGUF (provenance-documented conversion)

An F16 GGUF conversion of Qwen/Qwen3-Embedding-4B for serving with llama-server, published with a complete, verifiable provenance chain — source revision, converter version, and output checksum.

Why this repo exists

Qwen3-Embedding GGUFs have a known conversion-vintage trap: llama.cpp's Qwen3-Embedding conversion fixes (EOS/sep handling, pooling metadata — definitive fix #15023, 2025-08-02) live in convert_hf_to_gguf.py, so a GGUF converted before the fix has the bug baked in permanently — a current runtime cannot repair it (see #14234). Many third-party conversions have squashed histories, making their conversion vintage unprovable.

This conversion is documented at commit level so you don't have to trust — you can check:

The exact conversion command:

docker run --rm -v "$PWD/src:/src:ro" -v "$PWD/out:/out" \
  ghcr.io/ggml-org/llama.cpp:full-cuda13@sha256:500f241b5d7e27846bdda69ce6f4889db8de859ea3988d4265e5d9d7b7f1079b \
  --convert /src --outtype f16 --outfile /out/Qwen3-Embedding-4B-F16.gguf

Fidelity check

Verified against the official safetensors pipeline (sentence-transformers, same revision): cosine(reference, served) over a 66-probe multilingual set (Romanian, three diacritic-variant forms) — min cosine 0.99976, i.e. the served GGUF endpoint reproduces the reference embeddings.

Serving

llama-server -m Qwen3-Embedding-4B-F16.gguf --embeddings --pooling last -ngl 999 \
  --ctx-size 8192 --ubatch-size 2048

• --pooling last must be explicit. Qwen3-Embedding is a causal decoder — CLS/mean pooling produces silently wrong embeddings. Bring-up check: the verbose server log should print pooling type = 3, and any returned vector should have L2 norm ≈ 1.0 (normalization is server-side by default).
• Never combine --embeddings with --reranking on one instance (llama.cpp #20085: all-zero embeddings).
• VRAM guidance: F16 weights are ~8.0 GiB; on a 16 GiB card use --ubatch-size 2048 (a 8192 ubatch allocates a ~4.9 GiB compute buffer and OOMs). Ubatch is throughput-only for causal decoders — it never changes embedding values.

Query instruction protocol (client-side)

Per the official model card, queries should be instruction-formatted while documents are embedded plain:

Instruct: {task_description}
Query:{query}

The server embeds whatever text it receives — applying the instruction prefix is the caller's job. Skipping it typically costs 1–5% retrieval quality.

License & attribution

Apache-2.0, same as the base model. All model weights are © the Qwen team (Qwen/Qwen3-Embedding-4B); this repo contributes only the format conversion and its provenance documentation.