giga-embeddings-0826-3b-mlx-q8

Giga Embeddings 0826 3B — recommended MLX Q8 model for Apple Silicon

Русская карточка · All MLX models · GitHub · PyPI · Original paper

The recommended default for local Russian and English semantic search, RAG, text similarity, clustering, and classification on Apple Silicon. This mixed Q8/BF16 artifact keeps the embedding and final edge layers in BF16 while quantizing eligible inner weights to Q8.

These custom MLX encoder weights are loaded with giga-embeddings-mlx as shown below. Direct import into LM Studio or Ollama has not been verified. For those apps, see the corresponding 3B GGUF release and its compatibility notes.

Quality and size at a glance

• Original model quality: 74.56 Russian MTEB in the authors' paper.
• Our Q8 preservation check: aggregate NDCG@10 change +0.00181 versus the native MLX BF16 model—no measured aggregate regression on the frozen set.
• Q8 savings: download 6.312 → 3.755 GB (41% smaller); peak Metal memory 7.694 → 5.137 GB (33% lower) versus native MLX BF16.
• Typical time for one 512-token text: 0.637 s on an M4 Pro.

The official MTEB result belongs to the original BF16 model. We did not rerun the complete MTEB suite on Q8; our local test separately checks whether the MLX port and quantization preserve retrieval behavior.

Choose a released MLX model

Use

python -m pip install giga-embeddings-mlx

from giga_embeddings_mlx import load_embedding_model

model = load_embedding_model("default")
documents = model.encode_documents(["Москва — столица России."])
queries = model.encode_queries(
    ["Где находится Москва?"],
    instruction="Given a question, retrieve passages that answer the question",
)
scores = queries @ documents.T

Queries require an explicit instruction; documents have no prefix. Normal inference does not execute Python code from this model repository.

What is inside

• Affine Q8 inner weights, BF16 embedding and final edge layers, group size 64.
• Embedding dimension: 2048; maximum sequence length: 8192.
• Artifact size: 3.755 GB; peak Metal memory: 5.137 GB for 16 long texts.
• Batch speed: 0.73 documents/s for 16 texts of 1024 tokens each.
• Min/mean vector cosine versus native MLX BF16: 0.992840 / 0.999575.
• Top-1 agreement: 99.61%; mean top-10 overlap: 98.52%.
• RuSTS change: −0.000127; classification accuracy change: −0.000541.

Full methodology, speed samples, backend gates, and evidence hashes are in the 0826 MLX report. Q8 reduced disk and peak Metal memory, but was slower than BF16 in the displayed workloads.

Source and reproducibility

• Original model: ai-sage/Giga-Embeddings-instruct-3B-0826.
• Original paper: arXiv:2608.23806.
• Base revision: ed7db5c91b900b39381b27b6e9c0a3d31137cd29.
• Weight release: 0826-v0.1.0; tensor bytes are unchanged in the 0.1.2 documentation update.
• Converter commit: dfbc6a375ccdb637d1932529acbcfbf4db5025b6.
• manifest.json records the portable inventory and SHA-256 hashes.

Tested on Apple Silicon/macOS. This is an independent ai-babai quantization, not an official ai-sage release.

Citation

Please cite both this MLX software release and the original Giga-Embeddings work:

@software{popkov2026gigaembeddingsmlx,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings MLX},
  year    = {2026},
  version = {0.1.2},
  url     = {https://github.com/ai-babai/giga-embeddings-mlx}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}