bge-m3-query-clients

bge-m3 query-side clients (GGUF, 321–355 MiB, calibrated on Czech text)

Small query encoders for BAAI/bge-m3 (XLM-RoBERTa large, CLS pooling, 1024-d, 100+ languages; dense retrieval only). Use one of them wherever you already have a document index built with bge-m3's dense embeddings and want to encode queries on the client (laptop, browser tab, edge device): the index stays as it is, the query vectors stay compatible, and the file is 321–355 MiB instead of 1 104 MiB (fp16). The quantization is calibrated on Czech text; it holds equally on English (SciFact) because the encoder tolerates the grid well. Runs unchanged in llama.cpp (llama-embedding --pooling cls) and in the browser (wllama 3.6.1).

Half of a bge-m3 file is the 250 002 × 1024 token table; both files keep it in 4-bit (q4_0), which costs ≤ 0.2 nDCG points against an 8-bit table on this model. The file badge on Hugging Face shows the type of that token table (Q4_0); the block type (Q4_K_M, Q3_K) is in the file name and in the GGUF header (general.file_type 15 / 12).

Compatibility

Compatible with document vectors from BAAI/bge-m3 dense embeddings: CLS pooling, L2 normalisation, 1024 dimensions, no prompt on either side (bge-m3 uses none). Not compatible with bge-m3's sparse or ColBERT outputs, nor with indexes of any other model.

How to use

llama-embedding -m bge-m3-imx-Q4_K_M-generic_cs-tabq4_0.gguf -c 512 --pooling cls --embd-normalize 2 -p "<your query>"

Set the context to the query length (-c 512). In the browser wllama 3.6.1 loads the file directly (pooling cls).

Results

Test split, native llama-embedding, fp32 document index of the same model. nDCG@10 (% of fp32), mean cosine of the client's query vector to the fp32 query vector, mean top-10 overlap with the fp32 ranking.

Read differences under 0.01 nDCG@10 as ties (calibration-draw variance ~0.01, between-machine ±0.006). The Czech evaluation uses synthetic queries (Qwen3-1.7B doc2query, relevant = source segment), no human relevance judgements; its fp32 level is the same for every base model we tried (0.317–0.319), so it measures the compression, not the base.

Recipe

# llama-imatrix refuses encoders that append EOS: collect the matrix on a copy with the flag cleared
python gguf-py/gguf/scripts/gguf_set_metadata.py bge-m3-f16-noeos.gguf tokenizer.ggml.add_eos_token "" --force
llama-imatrix -m bge-m3-f16-noeos.gguf -f generic_cs.txt -c 512 -b 512 -ub 512 --chunks 585 -o imatrix.gguf
llama-quantize --imatrix imatrix.gguf --token-embedding-type q4_0 bge-m3-f16.gguf out.gguf Q4_K_M

generic_cs.txt = 3 000 paragraphs of Czech Wikipedia (sampling script in the repository). Verification harness and report: https://github.com/rosecky/embedding-quantization-public.

sha256: fe1011d5ec5cff995cb687f42afc47edcaf68cc8296dda11aeb7bc9640b8e295 (Q4_K_M + q4_0), b977fd649e0750fa8092970ff8bb8d8f955e684d9cbbef055bbe89061af38fdc (Q3_K + q4_0).

Everything behind these files is open (2026-09-16)

The complete pipeline is public under Apache-2.0 in https://github.com/rosecky/embedding-quantization-public: the verification recipe, the GPTQ exporter onto the llama.cpp K-quant grids, the llama-quantize recipes of these files, and since 2026-09-16 also the vector-quantisation side (quantiser, .vqw compiler, structured rotation, post-pass and the WebGPU runtime, client/vqweb). The public demo at https://thinletter.io/demo runs both the scalar files (wllama) and the vector-quantised containers. Questions and index-specific requests: [email protected], or the issues of the repository.

Licence

Weights derive from BAAI/bge-m3 (MIT) and are released under MIT. Calibration text: Czech Wikipedia (CC BY-SA; sampling script published, text not redistributed).