amaretto-embed-148m

amaretto-embed-148m

A 148M-parameter EmbeddingGemma, specialised for 8 Latin-script languages + code — a general-purpose drop-in at half the size and ≈40% of the RAM.

amaretto-embed-148m is google/embeddinggemma-300m with its vocabulary sliced from 262,144 → 60,497 tokens, and the resulting damage healed by distillation against the original model.

The insight: EmbeddingGemma spends two-thirds of its parameters (201M of 303M) on a token-embedding table sized for 100+ languages. If you only need a few languages, most of that table is dead weight. Cutting it down to the tokens that 8 Latin-script languages and code actually use halves the model — while leaving the transformer layers, where the understanding happens, completely untouched.

Result (real MTEB, controlled A/B vs EmbeddingGemma): ≥99.3% on retrieval and STS across all 8 languages, ≈98% on classification — the latter at the theoretical limit of a sliced vocabulary.

(Deployment footprint is architecture-determined; the RAM figure is for the identical-architecture f16 GGUF.)

Intended use — a general-purpose drop-in for EmbeddingGemma

If you run google/embeddinggemma-300m and only need these 8 languages, swap this in and halve your memory. It was distilled against EmbeddingGemma itself with absolute-vector MSE, which preserves the original embedding space (not merely the ranking), so its vectors mean what EmbeddingGemma's mean — the same task prefixes, 768 dims, Matryoshka sizes, and SentenceTransformer API.

Text embeddings — retrieval, semantic search, RAG, clustering, classification, STS — for English, Spanish, Portuguese, French, German, Italian, Dutch, Polish, and source code, especially where memory is tight (edge / Jetson / cheap CPU).

Re-index when you switch. This is a drop-in behaviourally, not bit-compatibly — at ≈0.98–0.99 cosine to the original the vectors are close but not identical, so don't query an existing EmbeddingGemma-built index with this model. Re-encode your corpus.

⚠️ Critical limitation: this model is NOT multilingual

The vocabulary for every non-target language was deleted. Text in Chinese, Japanese, Korean, Arabic, Hebrew, Cyrillic, Greek, Indic scripts, Thai, etc. will not error — it falls back to byte-level tokens — but sequences become drastically longer and embedding quality degrades severely. Do not use this model outside the 8 listed languages. For broad multilingual coverage, use the original google/embeddinggemma-300m.

Usage

Requires the same task prefixes as EmbeddingGemma — the model was trained with them.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("AmarettoLabs/amaretto-embed-148m")

query = model.encode("task: search result | query: how do I sort a list in python?")
docs  = model.encode([
    "title: none | text: sorted() returns a new sorted list from an iterable.",
    "title: none | text: De kat zat op de mat.",
])
from sentence_transformers.util import cos_sim
print(cos_sim(query, docs))

Matryoshka dimensions

Trained to preserve EmbeddingGemma's Matryoshka property at 768 / 512 / 256 / 128. Truncate and re-normalise:

import torch.nn.functional as F
emb_256 = F.normalize(embeddings[:, :256], p=2, dim=1)

Truncation is free at encode time — it only reduces storage and search cost. 256 is a strong default.

Other formats

Companion repos for running the model without PyTorch. Both are verified against this model's own embeddings, not against each other:

The ONNX builds are end-to-end graphs — pooling, both Dense heads, and normalization are inside the graph, so onnxruntime + tokenizers is the whole runtime dependency. That also means they are not the SentenceTransformer(..., backend="onnx") layout; for that workflow use this repo with PyTorch.

Versioning

Releases are tagged (e.g. v1.0.0). Pin a revision for reproducible embeddings — if you build a vector index with this model, load the same tag so future updates can't shift your vectors:

model = SentenceTransformer("AmarettoLabs/amaretto-embed-148m", revision="v1.0.0")

Loading without a revision tracks main, which may move. The GGUF and ONNX companion repos are versioned in lockstep.

How it was built

1. Vocabulary slicing (surgery, no training). Token frequencies were counted over Wikipedia (8 languages) + code. The keep-set is the top ≈60k by a balanced per-language mix, plus force-keeps: all special/control tokens, all 256 byte-fallback pieces, every single-char Latin-1 + Polish (ą ć ę ł ń ś ź ż) + æ piece, and the task-prompt tokens. Surviving embedding rows are index-selected, so kept tokens keep their exact original weights.

2. Distillation recovery. Slicing changes tokenization at the seams, which shifts the pooled embedding. The sliced model was trained to regress onto EmbeddingGemma's exact embedding vectors with an un-normalized Matryoshka MSE loss — which preserves the absolute embedding structure that linear-probe tasks (classification) depend on, not just the direction that retrieval needs.

• Data: 1.2M prefixed sentences — fresh Wikipedia (8 languages) + code, of which ≈21% are long documents (256–512 tokens) to support long-form retrieval — sampled from a region disjoint from the eval sets (no leakage).
• Full fine-tune, learning rate 5e-6 (a gentle rate that adapts the backbone just enough to keep retrieval/STS while preserving the class-separable structure), max_seq_len 512, 1 epoch. Warm-started from the untrained slice.

Evaluation

Measured on MTEB (v2), as a controlled A/B: both this model and EmbeddingGemma were run through the same MTEB version and are reported as % of EmbeddingGemma (never against externally-published numbers, which would confound library/prompt differences).

Retrieval — BelebeleRetrieval (nDCG@10)

Semantic similarity — STSBenchmarkMultilingualSTS (Spearman)

Classification — MassiveIntentClassification (accuracy)

On classification's ≈98%: this is the ceiling, not a shortfall. The vocabulary slicing itself — deleting ≈200k embedding rows — costs classification quality that no amount of training can recover: the untrained sliced model already floors at ≈97.7% of EmbeddingGemma on this task. amaretto's 97.9% is at that limit. Retrieval and STS depend on the relative geometry of the embedding space, which slicing barely perturbs (hence ≥99.3%); classification's linear probe depends on the absolute feature structure, which the reduced vocabulary coarsens. This is the physics of the compression, and the un-normalized-MSE objective was chosen specifically to get classification as close to that ceiling as possible.

Code retrieval — CodeSearchNetRetrieval (nDCG@10)

Natural-language docstring → code-function retrieval, across 6 programming languages (same controlled A/B):

98.7% of EmbeddingGemma on code retrieval. Note Ruby holds 99.6% even though it was not in amaretto's training code mix (python/java/javascript/php/go) — evidence the retained backbone generalizes beyond the trained languages. Java is the softest (96.3%).

Long documents — MLDR (MultiLongDocRetrieval, nDCG@10)

The model was trained with ≈21% long documents (256–512 tokens) to support long-form retrieval, and accepts the full 2048-token context. Measured on MLDR (genuine multi-thousand-token documents) for English and Spanish, same controlled A/B (both models, mteb 2.18.3, 2048-token context), reporting the test split:

Long-form retrieval is amaretto's softest axis — ≈96% of EmbeddingGemma on the test split, versus ≥99.3% on short retrieval and STS. Spanish holds up strongly (≈98%); English long-doc is the weak spot (≈94%). This is expected: long-form fidelity is the hardest thing to preserve through vocabulary slicing and distillation. The low absolute nDCG reflects MLDR's intrinsic difficulty (it's the same for the teacher) — the meaningful figure is the ratio to EmbeddingGemma. If long English documents are your primary workload, benchmark on your own data before switching.

Code and reproducibility

The full pipeline — vocabulary slicing, distillation training, the MTEB evaluation harness, per-model SHA-256 checksums, and the raw result JSONs behind every number above — is open at github.com/AmarettoLabs/amaretto-embed. You can re-derive the comparison tables from the committed results, or verify the model files you download here against the published checksums.

Limitations and bias

• Only the 8 listed languages + code. See the critical limitation above.
• Classification tops out at ≈98% of EmbeddingGemma — the vocabulary-slicing ceiling, not a training defect. Retrieval and STS are ≥99.3%.
• Long-form retrieval is softer (≈96% on MLDR test), and weakest for long English documents (≈94%). Short retrieval and STS remain ≥99.3%. Benchmark on your own data if long-document retrieval is your primary use case.
• Inherits EmbeddingGemma's biases and quality ceiling — distillation cannot exceed the teacher.
• Evaluated on BelebeleRetrieval, STSBenchmarkMultilingualSTS, MassiveIntentClassification, CodeSearchNetRetrieval, and MLDR (MultiLongDocRetrieval). Behaviour on other tasks/corpora may differ.

License — Gemma Terms of Use

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

amaretto-embed-148m is a Gemma Model Derivative of google/embeddinggemma-300m. Your use, reproduction, and distribution of it are governed by the Gemma Terms of Use, a copy of which is distributed with this model in the LICENSE file. By using this model you accept those terms.

Use restrictions (Gemma Terms §3.2) — binding on you

You may not use amaretto-embed-148m, or any model derivative of it, in violation of the Gemma Prohibited Use Policy, which is incorporated into the Gemma Terms of Use by reference and forms an enforceable condition of this model's license.

If you redistribute this model or a derivative of it

The Gemma Terms (§3.1) require you to:

1. Pass on the use restrictions (§3.2) as an enforceable provision, and notify recipients.
2. Give recipients a copy of the Gemma Agreement — ship the LICENSE file.
3. Mark modified files with prominent notices that you modified them.
4. Ship a NOTICE file containing exactly: "Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms" — see NOTICE.

You may add your own terms for your own modifications, provided they do not conflict with the Gemma Terms.

Modifications made to the original (Gemma Terms §3.1(c))

Full detail in NOTICE:

Trademarks (Gemma Terms §4.2)

This model is not affiliated with, endorsed by, or sponsored by Google. No rights to Google trademarks are granted. References to "Gemma" and "EmbeddingGemma" are descriptive, identifying the upstream model this work derives from.

Other rights

• Training data derives from CC-BY-SA Wikipedia text and from source code under various licenses. The corpus itself is not redistributed here.
• Google claims no rights in the embeddings you generate with this model (Gemma Terms §3.3).

Acknowledgements

Built on EmbeddingGemma by Google DeepMind.

Citation

@misc{amaretto_embed_148m,
  author = {{AmarettoLabs}},
  title  = {amaretto-embed-148m: a vocabulary-sliced, distillation-recovered EmbeddingGemma for 8 languages and code},
  year   = {2026},
  url    = {https://huggingface.co/AmarettoLabs/amaretto-embed-148m},
  note   = {Model derivative of google/embeddinggemma-300m},
}