strizh-ru-retriever-GGUF

strizh-ru-retriever — GGUF

GGUF builds of AGmind/strizh-ru-retriever — a tiny (24.4M), no-prefix RU-first dense retriever (384-dim, mean pooling; architecture accepts 8192 tokens, trained at 256). For architecture, training, the full evaluation grid and the honest positioning ("beats rubert-tiny2 on Russian and Vulkan throughput but is slower on CPU; below USER2-small/bge-m3 on quality; weak cross-lingual"), see the main model card.

Files

Just use Q8_0. The 50 368-token embedding table dominates the file and stays near 8-bit regardless, so Q5_0/Q4_0 save only ~2–3 MB while losing quality. K-quants (Q4_K / Q6_K) are unavailable — hidden 384 / intermediate 576 are not multiples of the K-quant super-block (256), so llama-quantize silently falls back to legacy quants.

Serve (llama.cpp — AMD / Vulkan / CPU)

llama-server -m strizh-ru-retriever.Q8_0.gguf \
  --embeddings --pooling mean -ngl 999 -c 8192 -b 4096 -ub 4096

--pooling mean is required — the default cls collapses recall for this model. The model is drop-in: encode queries and documents the same way, no prefixes.

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"input": "как включить mean-пулинг у эмбеддера"}'

License

Apache-2.0.