Downloads · 30 days
125
24% of all-time downloads
xero0000/Gemma-4-26B-A4B-it-vram14-iq4xs-imat
Gemma-4-26B-A4B-it-vram14-iq4xs-imat is a text generation model from xero0000. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
~13.3 GB all-VRAM mixed GGUF of google/gemma-4-26B-A4B-it (26B MoE, ~4B active), tuned so the full weight set fits dual mid-range GPUs and still beats the Q8 source on a coding/agent holdout.
Downloads · 30 days
125
24% of all-time downloads
All-time downloads
517
Public
Repo size
14.3 GB
Likes
1
Public
Click a slice to open those files.
.gguf14.3 GB · 100%
From the Hugging Face model README
~13.3 GB all-VRAM mixed GGUF of google/gemma-4-26B-A4B-it (26B MoE, ~4B active), tuned so the full weight set fits dual mid-range GPUs and still beats the Q8 source on a coding/agent holdout.
TL;DR: 4.52 bpw custom mix (experts IQ4_XS, attention Q5_K, dense FFN Q6_K) + imatrix. On a 3060 Ti 8 GB + 3080 10 GB desktop: ~3060 t/s prefill / ~98 t/s decode at 128K all-VRAM — about 13.5× prefill and 5.3× decode vs the same model’s Q8 production recipe (CPU-MoE offload). Perplexity is quality-neutral vs Q8 on the target corpus.
| File | Size | BPW | Notes |
|---|---|---|---|
gemma-4-26B-A4B-it-vram14-iq4xs-imat.gguf | 13.30 GiB | 4.52 | production daily driver |
Source was the official instruct Q8_0 GGUF (~25 GiB), requantized with an importance
matrix (--allow-requantize).
Gemma-4 26B-A4B GGUF layout (relevant for the recipe):
ffn_gate_up_exps + ffn_down_exps dominate size (~90% of Q8 weights)output.weight)ffn_down_exps has 704 columns → not 256-divisible; stock IQ4_XS fails there, so
that tensor uses IQ4_NL insteadThis arch currently needs a Gemma-4–capable llama.cpp build (TurboQuant / gemma4 fork
or equivalent). Plain older mainline binaries that predate gemma4 will not load it.
Built with llama-quantize (TurboQuant gemma4 tree), imatrix-guided:
| Tensor class | Type | Rationale |
|---|---|---|
ffn_gate_up_exps | IQ4_XS | size lever (experts dominate) |
ffn_down_exps | IQ4_NL | 704-col fallback (IQ4_XS not valid) |
attn_(q|k|v|output).weight | Q5_K | every-token attention path |
dense ffn_(up|gate|down).weight | Q6_K | shared / non-expert FFN |
ffn_gate_inp (router) | Q8_0 | routing is tiny and quality-critical |
| token embeddings | Q6_K | tied embd / output path |
Base ftype q4_K so custom --tensor-type rules apply. Approximate command:
llama-quantize --allow-requantize --imatrix gemma4-it.imatrix.gguf \
--token-embedding-type q6_K \
--tensor-type 'ffn_gate_up_exps=iq4_xs' \
--tensor-type 'ffn_down_exps=iq4_nl' \
--tensor-type 'attn_(q|k|v|output)\.weight=q5_K' \
--tensor-type 'ffn_(up|gate|down)\.weight=q6_K' \
--tensor-type 'ffn_gate_inp=q8_0' \
gemma-4-26B-A4B-it-Q8_0.gguf \
gemma-4-26B-A4B-it-vram14-iq4xs-imat.gguf \
q4_K
Importance matrix: computed on a local coding/agent calibration corpus (~90 chunks, same domain family as the holdout). Imatrix here acts as light domain adaptation as well as rounding guidance.
Holdout corpus, n_ctx=512, 24 chunks (same methodology as the local vram13 series).
Gemma PPL scale is not comparable to Qwen-family numbers (different tokenizer /
corpus fit) — only within-family deltas matter.
| Model | PPL | vs Q8 same-placement |
|---|---|---|
| Q8_0 (auto-fit placement) | 26.588 | baseline |
| Q8_0 (nc23 CPU-MoE offload) | 27.061 | placement alone ~+1.8% |
| vram14-iq4xs-imat (this file) | 25.954 | −2.4% |
| Q4_K_M-imat (stock, discarded) | 24.834 | −6.6% (larger / slower / not all-VRAM) |
Verdict on this holdout: 4-bit is quality-neutral to slightly better than Q8, because the imatrix + domain match more than offsets quantization noise. A stock Q4_K_M scored a bit lower PPL but was larger (15.6 GiB), needed CPU-MoE offload on this rig, and lost badly on runtime — it was deleted after the bench.
Rig: RTX 3060 Ti 8 GB + RTX 3080 10 GB (18 GB total), Ryzen 5950X, DDR4.
Engine: Gemma-4–capable llama.cpp (TurboQuant fork), q4_0 KV, flash-attn on.
Perf prompt ≈ 6K-token prefill + 160 decode.
| Config | Prefill | Decode | Checks |
|---|---|---|---|
| vram14 @128K all-VRAM (ts 44,56, ub 512) | 3061.5 t/s | 97.5 t/s | exact ✓ |
| vram14 @192K all-VRAM | 3038.6 t/s | 98.1 t/s | exact ✓ reasoning ✓ browser ✓ |
| Q8 @256K nc23 no-op-offload (production ref) | 226.2 t/s | 18.5 t/s | exact ✓ |
| vram14 @256K or ubatch 1024 @192K | — | — | OOM (GDN chunked compute buffer) |
vram14 vs Q8 production: ~13.5× prefill, ~5.3× decode.
| ctx | fit | slack (approx) | recommendation |
|---|---|---|---|
| 128K | all-VRAM | ~1.4 GiB | daily driver |
| 192K | all-VRAM | ~750 MiB | light desktop only |
| 256K | OOM | — | needs offload / more VRAM |
At razor-edge VRAM the dual-GPU pipeline-parallel compute buffer reserve (~1.3–1.9 GiB on CUDA0) can log a transient allocation failure; the server then retries without pipeline parallelism and recovers at the speeds above. That log line is expected and harmless on this class of rig when fully loaded.
Thinking-capable instruct model — leave thinking enabled for agent/tool loops
(official IT is trained to plan). Recommended sampling (Gemma defaults):
temp 1.0, top_k 64, top_p 0.95. Clients that force temp ≈ 0.2 make tool
loops pathologically deterministic.
./llama-server \
-m gemma-4-26B-A4B-it-vram14-iq4xs-imat.gguf \
--jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
--ctx-size 131072 --parallel 1 --n-gpu-layers 99 \
--split-mode layer --tensor-split 44,56 \
--batch-size 2048 --ubatch-size 512 \
--temp 1.0 --top-k 64 --top-p 0.95 \
--no-mmap --threads 8 --no-warmup \
--port 8000
Same as above with --ctx-size 196608. Expect ~750 MiB free after settle — fine for
interactive use, risky for long multi-tool sessions with a busy desktop compositor.
--tensor-split / use a single-card split.--n-cpu-moe / equivalent) and will lose
most of the all-VRAM speedup; prefer a smaller quant or more VRAM.Adjust --tensor-split for your card sizes (44,56 targets 8+10 GB).
| Base | google/gemma-4-26B-A4B-it (Google DeepMind) |
| Source GGUF | official instruct Q8_0 (Unsloth-labeled metadata in header) |
| Quant + imatrix + benches | xero0000, 2026-07-18 |
| Tooling | TurboQuant / gemma4 llama-quantize + llama-imatrix |
License: Apache-2.0 (same family as the base — Gemma 4 license). Quantization does not change the model license.