embeddinggemma-300m-8bit-mlx

embeddinggemma-300m — MLX 8-bit (compact)

An 8-bit (affine, group size 64) MLX quantization of google/embeddinggemma-300m, produced with mlx-embeddings. Runs the full EmbeddingGemma pipeline (mean-pool → 2× Dense → L2-normalize), 768-dim output, Matryoshka truncation to 512/256/128 supported.

Which one do I want? This model is the smallest option and is fine for most use — it costs about −0.2 pp STS / −0.5 pp nDCG vs full precision (see below). If you want zero measurable accuracy loss, use the -hi-mlx sibling or the llama.cpp ggml-org GGUF instead.

Usage

pip install mlx-embeddings

import mlx.core as mx
from mlx_embeddings.utils import load

model, tokenizer = load("janihal/embeddinggemma-300m-8bit-mlx")

# EmbeddingGemma prompt prefixes matter:
QUERY = "task: search result | query: "
DOC   = "title: none | text: "

texts = [QUERY + "What is the capital of France?",
         DOC   + "Paris is the capital and most populous city of France."]

enc = tokenizer.batch_encode_plus(texts, return_tensors="mlx", padding=True)
out = model(enc["input_ids"], attention_mask=enc["attention_mask"])
emb = out.text_embeds                    # (2, 768), already mean-pooled + L2-normalized
sim = float((emb[0] * emb[1]).sum())     # cosine similarity

Prompt templates for other tasks (from config_sentence_transformers.json): Retrieval-query → task: search result | query: · Retrieval-document → title: none | text: · STS → task: sentence similarity | query: · Classification → task: classification | query: · Clustering → task: clustering | query: .

How it was produced

python -m mlx_embeddings.convert \
  --hf-path google/embeddinggemma-300m \
  --mlx-path embeddinggemma-300m-q8 \
  -q --q-bits 8 --q-group-size 64 --q-mode affine

---

Evaluation

All numbers below are from a controlled comparison on one Apple M5 Max. Reference = the same model converted to MLX float32 (no quantization). Quantized candidates are scored against it on two public benchmarks and on raw embedding agreement.

• STS: STS-Benchmark test, 1379 pairs → Spearman(cosine, gold 0–5).
• Retrieval: NFCorpus (BEIR) test, 323 queries × 3633 docs → nDCG@10.
• mean cos vs FP32: mean cosine between this model's output vectors and the FP32 reference's, over all 6714 benchmark texts.
• Δ confidence intervals are paired bootstrap, 2000 resamples.

Accuracy

† The unsloth GGUF is built from Google's QAT checkpoint and omits the two Dense projection layers (converted without --sentence-transformers-dense-modules). Its output is the pooled backbone in a different vector space — hence ~0 cosine to the reference even though its STS score is fine. Not a like-for-like point.

Read: this compact model's accuracy loss is small but real and consistently negative (STS, nDCG, and embedding cosine all point the same way). The ggml-org GGUF and the -hi-mlx sibling are effectively lossless (Δ CIs cover 0).

Weight-space quantization error

Param-weighted relative RMSE of the de-quantized weights vs the FP32 base:

MLX affine-8bit and llama.cpp Q8_0 quantize the weights equally well. The end-to-end accuracy gap of this model comes almost entirely from 8-bit quantizing the token-embedding table (65 % of the parameters); keeping it in float precision — what the -hi-mlx sibling does — removes the gap.

Performance (Apple M5 Max, both stacks on the Metal GPU)

· llama.cpp latency measured through llama-server and includes a localhost HTTP round-trip. Its model load is ~3× faster than MLX (no Python import).

Read: on this machine MLX is ~1.8× faster for bulk embedding and ~1.3–1.5× faster per query than the GGUF, at roughly half the RAM. MLX throughput peaks around batch 16–32 (batch 64 drops to 220 docs/s from padding waste). Single-run numbers, ±10 %.

Compared to other embedding models

Same benchmark (STS-B test, NFCorpus test) and same Apple M5 Max. MLX rows run on MLX/Metal, GGUF rows on llama.cpp/Metal. docs/s = wall time to embed the 3633-document NFCorpus corpus; query latency is single-text, warm.

Read: only Qwen3-Embedding-4B clearly out-retrieves EmbeddingGemma-300m (nDCG 40.8 vs ~39) — at ~25× the embedding time and 4× the RAM. Qwen3-Embedding-0.6B is a stronger pure-similarity model (STS 91.3) but a weaker retriever (nDCG 36.7) and ~5× slower. Qwen3-VL-Embedding-2B (multimodal) is weaker than EmbeddingGemma on text on both axes. For retrieval / RAG at this size, EmbeddingGemma-300m is the best accuracy per byte and per second.

(Nemotron-3-Embed-1B GGUF did not produce usable embeddings through this llama.cpp build and is omitted.)

Caveats

• "FP32 reference" is the MLX implementation; the GGUFs run in llama.cpp, so a small cross-framework gap (~0.1–0.2 %) is folded into their numbers.
• One benchmark pair (STS + one retrieval set). Not a full MTEB run.
• Performance measured on M5 Max; ratios shift with hardware, batch size, and text length. llama.cpp likely has some tuning headroom (-fa, threads, ubatch).
• Loadable only with mlx-embeddings, not vanilla sentence-transformers.

---

License & attribution

Derived from google/embeddinggemma-300m (Google DeepMind) by post-training weight quantization only — no fine-tuning.

Use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy. This is a modified version of EmbeddingGemma; the same terms and use restrictions apply to this model and its outputs.

Quantization tooling: mlx-embeddings · MLX. GGUF baselines: ggml-org/embeddinggemma-300M-GGUF, unsloth/embeddinggemma-300M-GGUF.