Downloads · 30 days
1.4K
41% of all-time downloads
ai-babai/giga-embeddings-0826-3b-gguf
giga-embeddings-0826-3b-gguf is a sentence similarity model from ai-babai. Use it when you need a score for how close two texts are. It is set up for gguf. The card lists the license as mit.
Русская карточка · GGUF collection · Original model · Smaller 480M GGUF · Original paper
Downloads · 30 days
1.4K
41% of all-time downloads
All-time downloads
3.5K
Public
Repo size
28.4 GB
Likes
4
Trending 1
Click a slice to open those files.
.gguf14.2 GB · 100%
From the Hugging Face model README
Русская карточка · GGUF collection · Original model · Smaller 480M GGUF · Original paper
Local Russian and English text embeddings for semantic search, RAG, text
similarity, clustering, and classification with LM Studio, stock llama.cpp, and Ollama.
This repository contains one BF16 reference and three direct quantizations of
the bidirectional Giga Embeddings 3B 0826 model.
Embedding model only: use LM Studio or llama-server POST /v1/embeddings,
or Ollama POST /api/embed. This is not a chat or generative model.
The Python sentence-transformers package does not load these GGUF files directly.
Start with Q8_0. It is the recommended quality/size default. Choose
Q4_K_M only when download size and memory matter most; it is explicitly an
experimental lightweight option. BF16 is the high-precision reference. Q6_K is
included for research, but is not a better default than Q8_0 or Q4_K_M.
This is an independent ai-babai GGUF conversion, not an official ai-sage
release.

| Variant | Best for | File size | Saving vs BF16 | Metal memory | CUDA peak VRAM |
|---|---|---|---|---|---|
| BF16 | high-precision reference | 6.31 GB | — | 8.00 GB | not measured |
| Q8_0 | recommended default | 3.35 GB | 46.8% | 5.04 GB | 5.38 GB |
| Q6_K | research / owner review | 2.59 GB | 58.9% | 4.28 GB | 4.62 GB |
| Q4_K_M | lightweight / experimental | 1.96 GB | 68.9% | 3.65 GB | 3.99 GB |
Apple Silicon uses unified memory. Metal allocation and process RSS are
different views of the same shared memory and must not be added together. Peak
process RSS in the same Mac runs was 8.21 / 5.20 / 4.45 / 3.84 GB for BF16 /
Q8_0 / Q6_K / Q4_K_M. All capacities in this section use decimal GB
(1 GB = 10^9 bytes); measured MiB values were converted to GB.
SHA256SUMS · Machine-readable manifest
Download the recommended file:
hf download ai-babai/giga-embeddings-0826-3b-gguf \
giga-embeddings-0826-3b-q8_0.gguf \
--local-dir .
Run a recent stock llama.cpp server:
llama-server \
-m giga-embeddings-0826-3b-q8_0.gguf \
--embeddings \
-c 2048 -b 2048 -ub 2048 -np 1 \
--cache-type-k f32 --cache-type-v f32 \
--flash-attn auto -ngl 99 \
--host 127.0.0.1 --port 8080
Use -ngl 0 for CPU-only execution.
For retrieval, prepend an instruction to the query and embed documents as plain text:
curl http://127.0.0.1:8080/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "giga-embeddings-0826-3b-q8_0.gguf",
"input": [
"Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
"Москва — столица Российской Федерации.",
"Париж — столица Франции."
]
}'
Compare the returned normalized 2048-dimensional embeddings with cosine similarity (equivalent to their dot product after normalization).
For symmetric tasks such as semantic similarity or deduplication, use the same instruction for both sides or no instruction. The GGUF metadata selects the required mean pooling; do not replace it with CLS or last-token pooling.
In LM Studio 0.4.25+, download or import one of the updated GGUF files in
this repository. The refreshed general.name metadata makes LM Studio list it
as a Text Embedding model automatically. In the Developer tab, start the
local server and load the model. For a downloaded local GGUF, the CLI equivalent
is:
lms import -L --user-repo ai-babai/giga-embeddings-0826-3b-gguf -y giga-embeddings-0826-3b-q8_0.gguf
lms server start
lms ls
lms load 'paste-model-id-from-lms-ls-here' -y
Call the OpenAI-compatible embeddings API:
curl http://127.0.0.1:1234/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "<model-id-from-LM-Studio>",
"input": [
"Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
"Москва — столица Российской Федерации."
]
}'
Use the model ID shown by LM Studio; it depends on how the file was imported.
The output has 2048 dimensions. On LM Studio 0.4.25 Build 1, fresh imports of
all four updated files appeared as embedding models. For each variant,
POST /v1/embeddings returned normalized 2048D vectors with a related passage
closer to the query than an unrelated passage. These are functional smoke tests,
not new quality benchmarks.
If an older file or cached import still appears as an LLM, replace/re-import it. For a local copy that cannot be replaced yet, open My Models → model → Domain Control → Override Domain Type → Text Embedding. That manual override changes LM Studio's classification of the local model; it is not needed after a fresh import of the updated files on the tested version. Older LM Studio versions may need the override or a runtime update.
After downloading giga-embeddings-0826-3b-q8_0.gguf, create a Modelfile
next to it:
FROM ./giga-embeddings-0826-3b-q8_0.gguf
Import the embedding model and call the embeddings API:
ollama create giga-embeddings-0826-3b-q8 -f Modelfile
curl http://127.0.0.1:11434/api/embed \
-H 'Content-Type: application/json' \
-d '{
"model": "giga-embeddings-0826-3b-q8",
"input": [
"Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
"Москва — столица Российской Федерации."
]
}'
BF16, Q8_0, Q6_K, and Q4_K_M were compatibility-tested with Ollama 0.33.3 on Apple M4 Pro Metal. Each imports as a Qwen3 embedding-only model and returns finite, normalized 2048-dimensional vectors. This was a compatibility smoke test, not an Ollama quality or performance benchmark.
The original authors report 74.57 Russian MTEB, 71.93 English MTEB, 76.93 code MTEB, and 63.9 multilingual MTEB for the source BF16 model. Those numbers belong to the original model; we did not rerun the complete MTEB suites for these GGUF files.
Our giga-embeddings-external-retrieval-v1 evaluation used complete pinned
RuBQ (e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b) and SciFact
(d56462d0e63a25450459c4f213e49ffdb866f7f9) test splits, instruction-prefixed
queries, title + "\n" + text documents, and a 512-token contract. Values
below are equal-task macro averages. This is not a complete MTEB run, a
leaderboard submission, or a cross-model comparison.
| Variant | NDCG@10 | MRR@10 | Recall@10 | NDCG change vs BF16 |
|---|---|---|---|---|
| BF16 | 0.778758 | 0.767285 | 0.895762 | reference |
| Q8_0 | 0.778431 | 0.767302 | 0.893750 | −0.0327 points |
| Q6_K | 0.779334 | 0.768964 | 0.895146 | +0.0576 points¹ |
| Q4_K_M | 0.778297 | 0.769349 | 0.888652 | −0.0461 points |
¹The small positive aggregate difference is not evidence that Q6_K improves the model. A separate frozen representation comparison found one reproducible long-code outlier for Q6_K.
The stricter frozen multilingual/code holdout compared each quantized GGUF with the BF16 GGUF:
| Variant | Min / mean vector cosine | Top-1 agreement | Mean top-10 overlap |
|---|---|---|---|
| Q8_0 | 0.993540 / 0.999734 | 100.00% | 99.06% |
| Q6_K | 0.974783 / 0.997845 | 99.61% | 97.07% |
| Q4_K_M | 0.950085 / 0.982052 | 96.88% | 91.37% |
These agreement numbers measure preservation versus our BF16 GGUF, not absolute retrieval accuracy. All four files passed functional runtime validation. The larger representation drift is why Q4_K_M is labeled experimental even though its full RuBQ+SciFact NDCG loss was small; Q6_K's lower minimum comes from one reproducible long-code sample.
Median total throughput, after two warmups and across five repetitions, context 2048 and parallelism 1:
| Backend / workload | BF16 | Q8_0 | Q6_K | Q4_K_M |
|---|---|---|---|---|
| Apple M4 Pro Metal, 1×512 | 976 tok/s | 834 tok/s | 907 tok/s | 799 tok/s |
| Apple M4 Pro Metal, 16×1024 | 864 tok/s | 788 tok/s | 661 tok/s | 732 tok/s |
| Apple M4 Pro CPU, 1×512 | 243 tok/s | 345 tok/s | 149 tok/s | 189 tok/s |
| Apple M4 Pro CPU, 16×1024 | 298 tok/s | 275 tok/s | 140 tok/s | 205 tok/s |
| RTX PRO 4500 CUDA, 1×512 | not measured | 10,031 tok/s | 8,460 tok/s | 9,393 tok/s |
| RTX PRO 4500 CUDA, 16×1024 | not measured | 11,185 tok/s | 9,141 tok/s | 10,263 tok/s |
The Mac had an Apple M4 Pro and 48 GB unified memory. On Metal, BF16 was faster than every quantized variant in the measured matrix; quantization primarily saves storage and memory. Q6_K was also unusually slow on the tested CPU and CUDA paths.
Validation used clean stock ggml-org/llama.cpp commit
e750b887a82719c27200b71545f63ed78ec24719 (Linux build 10763).
| Runtime/backend | BF16 | Q8_0 | Q6_K | Q4_K_M |
|---|---|---|---|---|
| macOS Apple Silicon / Metal, CLI + server | tested | tested | tested | tested |
| macOS Apple Silicon / CPU, server resource API | tested | tested | tested | tested |
| Clean stock Linux / CUDA, CLI + server | tested | tested | tested | tested |
| Clean stock Linux x86 CPU, CLI + server | not tested | tested | not tested | tested |
| Ollama 0.33.3 / Apple Metal | tested | tested | tested | tested |
All tested Linux lanes passed llama-embedding, llama-server --embeddings,
single/batch/repeat/permutation checks, and finite unit-norm 2048-dimensional
output checks. A separate fresh server process reproduced the same eight exact
test embeddings on each tested Linux lane (cosine 1.0, maximum component delta
0).
llama.cpp/Ollama builds were not tested.| File | Bytes | SHA-256 |
|---|---|---|
giga-embeddings-0826-3b-bf16.gguf | 6,307,611,104 | d7093c8af0b02378552e4281f2f2e44bcd36ff35bb4553ebfba79ede2ccc4a9a |
giga-embeddings-0826-3b-q8_0.gguf | 3,354,067,424 | 8c75a39b6b91cb7e3e384de44cd01398a3876a641eb743b2127aba111896d6f9 |
giga-embeddings-0826-3b-q6_k.gguf | 2,591,068,640 | e66e22156d5858bff0f2c038ffa3aef31c7c996a77b7f3bb0c3984fdd172b6ba |
giga-embeddings-0826-3b-q4_k_m.gguf | 1,960,915,424 | 6107a9c6639d275c708fa227d5a3cfb285b2cc4b123ef39b1fecb760bd4ef5c0 |
ai-sage/Giga-Embeddings-instruct-3B-0826ed7db5c91b900b39381b27b6e9c0a3d31137cd29model.safetensors SHA-256:
de8519bef7ee360043970b0081088c5294e2a9196ad2d0570a33c4f51bb2e134tokenizer.json SHA-256:
6fb1280bd7fd529f425929b5df823a5a44485cd7fa9679d1ec1acaac4962e8catokenizer_config.json SHA-256:
843eeba481465c1485a5b5f24bd24d6c12c4e502c16f093c3ab6a0f058c2c5f2409723a88b12071974ed5924a2dc1c8b2b2064f7 on top of upstream llama.cpp
e750b887a82719c27200b71545f63ed78ec24719ggml-org/llama.cpp@e750b887a82719c27200b71545f63ed78ec24719, build 107637ba4e00e76b27316b1b3709476d807958bcd1e9dMachine-readable provenance and hashes are also available in manifest.json
and SHA256SUMS. The current GGUF revision changes descriptive metadata only;
tensor payloads remain byte-identical to the prior release.
Локальная GGUF-версия Giga Embeddings 0826 3B для семантического поиска, RAG, сравнения текстов, кластеризации и классификации на русском и английском языках. Полное описание на русском.
Please cite both this GGUF release and the original Giga-Embeddings work:
@software{popkov2026gigaembeddingsgguf,
author = {Maksim Popkov},
title = {Giga Embeddings 0826 GGUF},
year = {2026},
url = {https://huggingface.co/ai-babai/giga-embeddings-0826-3b-gguf}
}
@misc{kolodin2026gigaembeddings,
title = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
author = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
year = {2026},
eprint = {2608.23806},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.23806}
}