bekko-embedding-v1-a8m-GGUF

bekko-embedding-v1-a8m-GGUF

GGUF release of hotchpotch/bekko-embedding-v1-a8m, an ultra-compact multilingual text embedding model with 7.7M active parameters.

llama.cpp

The GGUF contains a compatible Gemma2 SentencePiece vocabulary and runs on the standard llama.cpp runtime with ModernBERT embedding support:

# GPU or Apple Silicon: use BF16
llama-server \
  -hf hotchpotch/bekko-embedding-v1-a8m-GGUF:BF16 \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bekko","input":["What is the tallest mountain in Japan?","Mount Fuji is the tallest mountain in Japan."]}'

For CPU inference, replace :BF16 with :Q8_0. If the selector is unavailable in your llama.cpp version, download the desired GGUF file and pass it with --model:

hf download \
  hotchpotch/bekko-embedding-v1-a8m-GGUF \
  bekko-embedding-v1-a8m-Q8_0.gguf \
  --local-dir .

llama-server \
  --model ./bekko-embedding-v1-a8m-Q8_0.gguf \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

Ollama

The default Ollama tag is BF16 and is recommended for GPU and Apple Silicon users:

ollama pull hotchpotch/bekko-embedding-v1-a8m
curl http://localhost:11434/api/embed \
  -d '{"model":"hotchpotch/bekko-embedding-v1-a8m","input":"What is the tallest mountain in Japan?"}'

For CPU inference, select the Q8_0 tag explicitly:

ollama pull hotchpotch/bekko-embedding-v1-a8m:q8_0

The available tags are latest (BF16), bf16, f16, and q8_0. The registry artifact has been verified as an embedding model with 384 dimensional, L2-normalized output.

Model contract

• Architecture: ModernBERT encoder
• Context length: 8192 tokens
• Embedding dimension: 384
• Matryoshka dimensions: 256, 128, 64
• Pooling: mean
• Similarity: cosine or dot product after L2 normalization
• Query/document prefixes: none
• Languages: 100+

Use the same unprefixed encoding function for queries and documents.

Quantization quality

Q8_0 was compared with the revision-pinned SentenceTransformers source using the same unprefixed queries and documents:

The mean cosine similarity to the HF embeddings is 0.999695 over five ordinary multilingual samples. Small positive retrieval deltas should be treated as ranking noise, not as an improvement caused by quantization.

Faster CPU inference

GGUF is intended for portable llama.cpp and Ollama deployment. On a fast x86 CPU, the optimized OpenVINO export is generally faster than llama.cpp GGUF and should be preferred when maximum throughput is the priority. It is available in the hotchpotch/bekko-embedding-v1-a8m source repository when maximum throughput is the priority. The source model also provides the original SentenceTransformers model and its full model card.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="openvino",
)
embeddings = model.encode(["What is the tallest mountain in Japan?"], normalize_embeddings=True)

Choosing a GGUF format

The repository publishes BF16, F16, and Q8_0. Use BF16 on GPUs and Apple Silicon; use Q8_0 on CPUs, especially CPUs without native BF16 arithmetic. F16 is provided for runtimes or hardware that prefer FP16 explicitly.

Q8_0 remains the best quantized portable trade-off found in our BF16/Q8_0/Q6_K/Q5_K_M/Q4_K_M/IQ4_XS comparison:

• Lower-bit K-quants provide almost no size reduction because the 384-wide tensors fall back to Q5_0 or Q8_0. Q4_K_M is only 1.9% smaller than Q8_0.
• Lower-bit embeddings have larger cosine drift. IQ4_XS loses 0.0195 mean NDCG@10 on the representative NanoCoIR tasks.
• Q8_0 is the fastest quality-preserving quantization on Raspberry Pi 5. Q4_K_M and Q5_K_M reach only 41% and 39% of Q8_0 throughput respectively.
• Q6_K has the same size, embeddings, benchmark scores, and effectively the same fallback tensor types as Q8_0, so publishing both would be redundant.

Positive benchmark deltas in this small evaluation are ranking noise rather than evidence that lower numerical precision improves the model.

Files and reproducibility

This repository is generated from an immutable source revision. See conversion.json for the exact revision, llama.cpp revision, SHA-256 hash, and release-critical config validation.

For benchmark results, training details, license, and limitations, refer to the source model card.