Downloads · 30 days
235
10% of all-time downloads
Krasnopjorovs/gemma-4-31b-it-Imatrix-GGUF
gemma-4-31b-it-Imatrix-GGUF is a text generation model from Krasnopjorovs. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
Verified quants only. Sub-4-bit variants are excluded from this release because they produce degenerate output on this model size — no point shipping broken files.
Downloads · 30 days
235
10% of all-time downloads
All-time downloads
2.3K
Public
Repo size
239 GB
Likes
0
Public
Click a slice to open those files.
.gguf239 GB · 100%
From the Hugging Face model README
Verified quants only. Sub-4-bit variants are excluded from this release because they produce degenerate output on this model size — no point shipping broken files.
GGUF imatrix builds of google/gemma-4-31B-it.
Built with llama.cpp and importance-matrix calibration on a public multilingual + code + math corpus. Every quant is loaded, prompted, and visually checked before publication.
Hi local LLM enthusiasts! You may have noticed a few quants from me over the past months — those were one-off experiments. This release is the first one from a fully automated pipeline.
Please try them out, share feedback, and let me know which quants you find useful or which ones you wish I had made. For now, anything below 4-bit produces degraded output on this architecture, so I am holding off on those until they actually work well. More quants and more models coming.
| Quant | Size | What it's for |
|---|---|---|
Q8_0 | 31G | Almost the original. Pick this if RAM isn't a concern. |
Q6_K_L | 24G | Near-lossless with Q8_0 embeddings. Best of the K-quants. |
Q6_K | 24G | Near-lossless. Excellent fidelity at smaller size than Q8_0. |
Q5_K_L | 21G | Q5_K_M with Q8_0 embeddings. High quality, small overhead. |
Q5_K_M | 21G | Sweet spot between Q4 and Q6. Solid all-rounder. |
Q5_K_S | 20G | Slightly smaller than Q5_K_M, virtually identical output. |
Q4_K_L | 18G | Q4_K_M with Q8_0 embeddings. The smart 4-bit choice. |
Q4_K_M | 18G | The default. Most-downloaded quant for a reason. |
Q4_K_S | 17G | A bit smaller than Q4_K_M, a tiny step down. |
IQ4_NL | 17G | ARM-optimized 4-bit. For Raspberry Pi & friends. |
IQ4_XS | 16G | Tightest 4-bit format. Quality close to Q4_K_S, smaller. |
*_L variants override the output tensor and embeddings to Q8_0 — small disk cost, better output stability.
Tl;dr:
<bos><start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
Single file (recommended):
hf download Krasnopjorovs/gemma-4-31b-it-Imatrix-GGUF \
--include "gemma-4-31b-it-Q4_K_M.gguf" --local-dir .
Whole repo:
hf download Krasnopjorovs/gemma-4-31b-it-Imatrix-GGUF --local-dir ./gemma-4-31b-it-gguf
./llama-server \
-m gemma-4-31b-it-Q4_K_M.gguf \
-c 32768 -ngl 99 \
--host 0.0.0.0 --port 8080
Then point your favorite chat client at http://localhost:8080/v1.
Imatrix generated from reapmix — community calibration mix (~400K tokens, multilingual + code + math). Same class of public calibration data used by other community publishers; this release makes no unique calibration claim.
The output tensor and embeddings carry disproportionate weight in quantized output. The *_L variants keep these at Q8_0 — small disk cost for noticeably better stability at low bit-rates.
google/gemma-4-31B-it