colibri-embed-ptbr

<div align="center">

🐦 Colibri — Brazilian-Portuguese Embeddings

A tiny embedding model that punches far above its weight.

Named after the colibri (hummingbird) — the smallest bird, yet it out-flies much larger ones.

~157M params · 768-dim · plain SentenceTransformer (no adapters) · runs on a cheap CPU VPS

</div>

---

Colibri is a compact Brazilian-Portuguese text-embedding model derived from google/embeddinggemma-300m. Despite being half the size, it matches or beats much larger multilingual embedders on MTEB-BR — including 7B and 27B models — and is designed small on purpose so it runs comfortably on a simple CPU VPS, with no GPU required.

Use cases: Portuguese semantic search, retrieval-augmented generation (RAG), retrieval, reranking, clustering, semantic textual similarity (STS) and classification — for Brazilian-Portuguese text across legal, tax, financial, medical, academic and general domains.

✨ Highlights

• 🏆 Beats bigger models on MTEB-BR — including embeddinggemma-300m, and 7B / 27B multilingual embedders.
• 🪶 Half the footprint of embeddinggemma-300m: ~607 MB vs ~1.2 GB, and less RAM.
• 💻 Runs on a $-few/month CPU VPS — benchmarked on 4 vCPU / 16 GB (no GPU).
• 🔌 Plain SentenceTransformer — no adapters, no LoRA at inference. Drop-in.
• 📐 Matryoshka dimensions (768 / 512 / 256 / 128) + an fp16 branch + a fast ONNX path.

---

🥇 Quality — MTEB-BR

Evaluated on MTEB-BR (22 native Brazilian-Portuguese tasks: retrieval, reranking, STS, classification, clustering, pair-classification). Score = mean over the 22 tasks.

A ~157M-parameter model outperforming 7B–27B multilingual embedders on Brazilian Portuguese.

Size ↔ quality frontier. Colibri sits on the open-model Pareto frontier for MTEB-BR — it dominates its own base embeddinggemma-300m (half the size, higher score) and matches models up to ~10× larger:

---

💻 Built for a simple CPU VPS

Colibri was designed to be small enough for cheap, GPU-less hosting. Benchmarked head-to-head against google/embeddinggemma-300m on a Verda CPU.4V.16G (4 vCPU, 16 GB RAM) — a typical small VPS — encoding Brazilian-Portuguese sentences:

Takeaways

• Half the size and less RAM than embeddinggemma-300m — fits a smaller, cheaper VPS.
• Same encode latency at equal precision (the vocabulary trim shrinks the token-embedding matrix, not the transformer compute) — you lose nothing in speed.
• The ONNX backend roughly doubles throughput (33 ms vs 76 ms) — best when latency matters; fp32 is the most RAM-frugal.

Benchmark hardware: Verda CPU.4V.16G. Numbers are indicative single-node CPU measurements.

---

🚀 Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("tardellirs/colibri-embed-ptbr")

query = "Como declarar imposto de renda de aluguel?"
docs = [
    "Rendimentos de aluguel devem ser informados na ficha de Rendimentos Tributáveis...",
    "O Pix é um meio de pagamento instantâneo criado pelo Banco Central...",
]

q = model.encode(query, prompt_name="query")      # "task: search result | query: "
d = model.encode(docs, prompt_name="document")    # "title: none | text: "
print(model.similarity(q, d))

Faster on CPU — ONNX backend (≈2× throughput, no code changes to your pipeline):

model = SentenceTransformer("tardellirs/colibri-embed-ptbr", backend="onnx")

Prompts follow EmbeddingGemma conventions (query / document / classification / clustering / similarity).

---

🎛️ Variants: precision & dimension

Precision — a half-precision branch (same quality, half the size; best for GPU/storage — on CPU it's not faster):

Matryoshka dimensions — truncate the 768-dim output to 512 / 256 / 128 for smaller vectors (cheaper index, faster search) with a graceful quality trade-off:

model = SentenceTransformer("tardellirs/colibri-embed-ptbr", truncate_dim=256)

(The MTEB-BR score above is for the full 768-dim embeddings.)

---

🔧 How it was built

A three-stage pipeline: vocabulary trimming → knowledge distillation → model soup.

1. Vocabulary trimming. tardellirs/embeddinggemma-pt-br trims embeddinggemma-300m's 262k multilingual vocabulary down to a ~64k Brazilian-Portuguese vocabulary, cutting the model from ~300M to ~157M effective parameters with negligible quality loss. Code: github.com/tardellirs/embedding-vocab-trimmer.

2. Knowledge distillation. Relational (similarity-preserving) KD from two complementary teachers — Qwen/Qwen3-Embedding-4B (strong at clustering) and Qwen/Qwen3-Embedding-8B (strong at retrieval / reranking). The student learns to reproduce the average of the two teachers' pairwise-similarity matrices (dimension-agnostic; preserves STS). Trained on a diverse ~100k-passage corpus of native Brazilian Portuguese spanning legal, tax, financial, medical, academic, technical, encyclopedic and general-web domains:

Evaluation integrity: the model is trained only on training / non-evaluation splits — every MTEB-BR evaluation example is held out, so the scores reflect generalization, not memorization.

3. Model soup + merge. Distillation checkpoints were combined (model soup / weight averaging) and linearly merged with the trimmed base, with the mixing weight chosen on held-out validation. Both operations yield ordinary weights, so the published model is a single standalone encoder.

---

🔗 Links

• Base: google/embeddinggemma-300m · Trimmed base: tardellirs/embeddinggemma-pt-br
• Trimming code: github.com/tardellirs/embedding-vocab-trimmer
• Distillation teachers: Qwen/Qwen3-Embedding-4B · Qwen/Qwen3-Embedding-8B
• Benchmark: MTEB-BR · CPU benchmark hardware: Verda

---

🙏 Acknowledgments

We gratefully acknowledge Verda for the GPU compute credits that supported this work.

<div align="center"> Built with vocabulary trimming · relational knowledge distillation · model soup. </div>