Downloads · 30 days
295
17% of all-time downloads
zhiqian99/Qwen3-Embedding-0.6B-GGUF
Qwen3-Embedding-0.6B-GGUF is a sentence similarity model from zhiqian99. Use it when you need a score for how close two texts are. It is set up for gguf. The card lists the license as apache-2.0.
All-in-one GGUF quantizations of Qwen/Qwen3-Embedding-0.6B, from 8-bit down to 1-bit, with importance-matrix calibration optimized for financial and technical text retrieval.
Downloads · 30 days
295
17% of all-time downloads
All-time downloads
1.8K
Public
Repo size
11.2 GB
Likes
0
Public
Click a slice to open those files.
.gguf11.2 GB · 100%
From the Hugging Face model README
All-in-one GGUF quantizations of Qwen/Qwen3-Embedding-0.6B, from 8-bit down to 1-bit, with importance-matrix calibration optimized for financial and technical text retrieval.
Qwen3-Embedding-0.6B is a compact, multilingual embedding model well suited for RAG pipelines, semantic search, and document retrieval. These quantizations make it practical to run on edge devices, laptops, and resource-constrained servers -- particularly for financial NLP workloads where low latency and small memory footprint matter.
The importance matrix was calibrated on a mixed corpus weighted toward financial data (financial Q&A from FiQA, SEC 10-K filings from FinanceBench, financial sentiment from Twitter, RAG pairs) alongside math reasoning and general text, so the quantized models preserve the weights most relevant to financial domain embeddings.
| Property | Value |
|---|---|
| Base model | Qwen/Qwen3-Embedding-0.6B |
| Parameters | 595,776,512 |
| Max context | 32,768 tokens |
| Pooling | Last token |
| Embedding dim | 1024 |
| License | Apache 2.0 |
| Quantized with | llama.cpp |
Despite their size, neural networks are remarkably sparse in information density. Most of the 16 bits allocated per weight during training exist to make gradient descent work -- not to store knowledge. Current estimates put the actual information content at roughly 2 bits per parameter. The remaining 14 bits are redundancy.
This explains why aggressive quantization works: compressing from 16-bit to 4-bit (75% reduction) discards almost exclusively noise. Our benchmark data confirms this -- Q3_K_M-imat at 4.66 BPW scores within +0.62 PPL of the full BF16 baseline while being 70% smaller than BF16 (331 MB vs 1.1 GB). For comparison, Q4_K_M-imat at 5.32 BPW shows +0.65 PPL -- the 3-bit model actually edges it out. The imatrix (importance matrix) is key here: by profiling which weights actually carry signal, we preserve those at higher precision and compress the rest. This is why imatrix-calibrated 3-bit models can outperform naive 5-bit quantizations.
The quality cliff appears around 3 BPW, where we start cutting into real information. Below that, PPL degrades rapidly (Q2_K at 3.97 BPW: +391, IQ1_S at 2.79 BPW: +23,008). Ternary quantizations (TQ2_0, TQ1_0) diverge entirely on this architecture.
All models evaluated with llama-perplexity on a 22 MB calibration corpus (financial, math, and general text). Context window: 1536 tokens. Chunks: 200. Lower PPL = better.
Baseline PPL (BF16): 406.0250
Notes:
| Use case | Model | Notes |
|---|---|---|
| Maximum quality | Q8_0 | Near-lossless, 610 MB |
| Best quality/size trade-off | Q3_K_M-imat | +0.62 PPL delta at 331 MB -- smallest model with near-baseline quality |
| Larger but safe margin | Q4_K_M-imat | +0.65 PPL delta at 378 MB |
| Extreme compression | Q2_K-imat | Usable for non-critical applications |
All models were quantized from the BF16 source using llama-quantize from llama.cpp.
Three strategies were used:
The importance matrix was generated from a mixed-domain corpus (22 MB, ~198,000 lines). The mix was chosen to cover the primary use case (financial text) while including general and mathematical text to maintain broad capability:
| Dataset | Source | Domain | Entries |
|---|---|---|---|
| WikiText-2 | ggml-org | General knowledge | 36,718 lines |
| Twitter Financial News | zeroshot/twitter-financial-news-sentiment | Financial sentiment | 9,543 |
| GSM8K | openai/gsm8k | Math word problems | 7,473 |
| Financial RAG | philschmid/finanical-rag-embedding-dataset | Financial Q&A pairs | 6,998 |
| FiQA | explodinggradients/fiqa | Personal finance Q&A | 5,650 |
| MATH Competition | DigitalLearningGmbH/MATH-lighteval | Competition math | 5,000 |
| FinanceBench | PatronusAI/financebench | SEC 10-K filings | 150 |
The financial datasets (FiQA + Twitter Financial News + Financial RAG + FinanceBench) contribute ~22,300 entries of domain-specific text covering sentiment, Q&A, RAG pairs, and SEC filings -- ensuring the importance matrix prioritizes weights relevant to financial terminology and reasoning.
./llama-server \
-m Qwen3-Embedding-0.6B-Q3_K_M-imat.gguf \
--embedding --pooling last \
-c 32768 -np 8 \
--host 0.0.0.0 --port 8080
from llama_cpp import Llama
model = Llama(
model_path="Qwen3-Embedding-0.6B-Q3_K_M-imat.gguf",
embedding=True,
pooling_type="last",
n_ctx=32768,
)
result = model.create_embedding(["Financial analysis of Q3 earnings"])
print(len(result["data"][0]["embedding"])) # 1024
from huggingface_hub import hf_hub_download
path = hf_hub_download(
repo_id="PeterAM4/Qwen3-Embedding-0.6B-GGUF",
filename="Qwen3-Embedding-0.6B-Q3_K_M-imat.gguf",
)
Apache 2.0, inherited from the original model.