gemma4-31b-imatrix-mtp-GGUF

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #99f6e4; border-radius: 16px; box-shadow: 0 10px 15px -3px rgba(0,0,0,0.05), 0 4px 6px -2px rgba(0,0,0,0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;"> <div style="background: linear-gradient(135deg, #0d9488 0%, #134e4a 100%); padding: 24px; color: white;"> <div style="display: flex; align-items: center; justify-content: space-between; flex-wrap: wrap; gap: 10px;"> <h1 style="margin: 0; font-size: 26px; font-weight: 800; display: flex; align-items: center; gap: 12px; color: white; border: none;">🧊 Google/Gemma-4-31B-it · imatrix · GGUF</h1> <span style="background: #f59e0b; color: #1c1917; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; text-transform: uppercase; letter-spacing: 0.5px;">imatrix (hybrid)</span> </div> </div> <div style="display: flex; gap: 8px; flex-wrap: wrap; padding: 12px 24px; background: #f8fafc; border-bottom: 1px solid #e2e8f0;"> <span style="background: #dbeafe; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #bfdbfe;">📦 10.2 · 13.4 · 15.6 · 19.8 GiB</span> <span style="background: #d1fae5; color: #065f46; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #a7f3d0;">IQ2_M · IQ3_M · IQ4_XS · Q5_K_S</span> <span style="background: #fef3c7; color: #92400e; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #fde68a;">🧪 2-bit AWQ · +54% tool-arg accuracy</span> <span style="background: #fce7f3; color: #9d174d; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #fbcfe8;">🏗️ llama.cpp f3e1828</span> <span style="background: #fef3c7; color: #92400e; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #fde68a;">🏅 Agent · 100% patch</span> <span style="background: #ede9fe; color: #5b21b6; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #ddd6fe;">👁️ Text + Image · mmproj 772 MB</span> <span style="background: #cffafe; color: #155e75; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #a5f3fc;">⚡ MTP drafter · 88% accept @ n=1</span> <span style="background: #dcfce7; color: #166534; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 20px; border: 1px solid #bbf7d0;">🎯 Q5_K_S · text+vision+MTP in 24 GB</span> </div> <div style="padding: 24px; display: flex; flex-direction: column; gap: 20px;"> <div style="background: #f0fdfa; border-left: 5px solid #0d9488; padding: 16px; border-radius: 0 8px 8px 0;"> <h3 style="margin: 0 0 8px 0; font-size: 15px; color: #115e59; font-weight: 700; display: flex; align-items: center; gap: 6px;"><span>🧊</span> What this is</h3> <p style="margin: 0; font-size: 13px; color: #334155; line-height: 1.7;">An aggressively compressed (under 3 bpw) <b>IQ2_M</b> quantization of <a href="https://huggingface.co/google/gemma-4-31B-it"><b>google/gemma-4-31B-it</b></a>, calibrated from real coding/tool-use logs. The 2-bit ships an <b>AWQ build</b> (activation-aware scaling) — chosen over plain imatrix by a direct tool-call test: <b>+54% tool-argument accuracy and far more consistent</b> at the same 2.85 bpw, because at 2 bits static KLD/PPL doesn't predict whether the model still fills correct tool arguments (see <a href="#-why-the-2-bit-is-awq">Why the 2-bit is AWQ</a>). Runs in vanilla <code>llama.cpp</code> / Ollama / LM Studio — <b>no custom runtime, no extra inference cost</b>. Higher-bit <b>IQ3_M</b> (3.76 bpw), <b>IQ4_XS</b> (4.36 bpw), and <b>Q5_K_S</b> (5.55 bpw — highest fidelity, KLD 0.025) builds ship plain <b>hybrid imatrix</b> for users with more VRAM.</p> </div> <div style="background: #faf5ff; border-left: 5px solid #7c3aed; padding: 16px; border-radius: 0 8px 8px 0;"> <h3 style="margin: 0 0 8px 0; font-size: 15px; color: #5b21b6; font-weight: 700; display: flex; align-items: center; gap: 6px;"><span>👁️</span> Now with vision (text + image input)</h3> <p style="margin: 0; font-size: 13px; color: #334155; line-height: 1.7;">Gemma 4 is natively multimodal. This repo ships the model's <b>vision tower as a separate <code>mmproj-gemma-4-31B-it-Q8_0.gguf</code></b> (772 MB, SigLIP-style 27-layer encoder, Q8_0 — visually lossless vs F16 at ⅔ the size). Pair it with <b>any</b> of the four quant files via <code>--mmproj</code> and the model can <b>see images</b> — describe screenshots, read diagrams, answer questions about a UI, and so on. The text quant is unchanged; vision adds only the small mmproj. See <b>Usage → Vision</b> below.</p> </div> <div style="background: #f0fdf4; border-left: 5px solid #16a34a; padding: 16px; border-radius: 0 8px 8px 0;"> <h3 style="margin: 0 0 8px 0; font-size: 15px; color: #166534; font-weight: 700; display: flex; align-items: center; gap: 6px;"><span>🎯</span> The 24 GB build (Q5_K_S)</h3> <p style="margin: 0; font-size: 13px; color: #334155; line-height: 1.7;">The new <b>Q5_K_S</b> build (5.55 bpw, <b>19.85 GiB</b>) is sized so a single <b>24 GB</b> GPU can host the <b>full stack at once</b>: the 5-bit text trunk (19.85) + the <b>Q4_K_M MTP drafter</b> (0.36) + the <b>Q8 vision mmproj</b> (0.75) = <b>~21.0 GiB</b>, leaving <b>~3.0 GiB</b> for a real KV cache (more with <code>--cache-type-k q8_0 --cache-type-v q8_0</code>). Near-FP16 quality (KLD 0.025), images, <b>and</b> speculative decoding — all on one consumer card, no offload.</p> </div> <div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 15px; margin-top: 10px;"> <div style="border: 1px solid #e2e8f0; padding: 14px; border-radius: 8px; background: #fafafa; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"><span style="font-weight: 700; color: #115e59; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">📉 ~5.6× smaller</span><span style="font-size: 13px; color: #4b5563; line-height: 1.5;">10.17 GiB on disk vs 57.2 GiB FP16, at ~2.85 bits/weight.</span></div> <div style="border: 1px solid #e2e8f0; padding: 14px; border-radius: 8px; background: #fafafa; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"><span style="font-weight: 700; color: #115e59; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">🤖 Actually agentic</span><span style="font-size: 13px; color: #4b5563; line-height: 1.5;">47% pass / 100% patch on a 10-instance agentic SWE-rebench holdout (IQ4_XS). IQ2_M still resolved 40% — best of every sub-3-bpw arm tested.</span></div> <div style="border: 1px solid #e2e8f0; padding: 14px; border-radius: 8px; background: #fafafa; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"><span style="font-weight: 700; color: #115e59; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">🛠️ Standard GGUF</span><span style="font-size: 13px; color: #4b5563; line-height: 1.5;">Loads anywhere llama.cpp runs. No patches, kernels, or forks.</span></div> </div> </div> </div>

📊 Unified benchmark & quality table

Agentic metrics from a SWE-rebench holdout run through the OpenAI Agents SDK (10 instances × 3 reps). Static metrics (PPL / KLD / top-p) measured against FP16 on a held-out eval corpus at ctx=4096. KLD column is median for robustness to per-token tails.

† IQ2_M now ships an AWQ build (see § Why the 2-bit is AWQ below). The SWE-rebench figures (†) were measured on the earlier imatrix 2-bit; the AWQ swap is justified by a direct tool-call fidelity comparison (param-acc +54%, far tighter run-to-run variance) at the same 2.85 bpw. AWQ also cuts IQ2_M perplexity nearly in half (1959 → 1040) — it trades a hair of median-KLD for a much more usable 2-bit.

Q5_K_S resolves 40% of the holdout (tying the old IQ2_M, ahead of IQ3_M) at 100% patch and a low 11% tool-error rate; IQ4_XS remains the agentic leader at 47% (gap within run-to-run noise).

<details> <summary><b>📌 Sampling & methodology details</b></summary>

Sampling: temperature=0.25, top_p=0.95, top_k=20, max_tokens=32768, ctx=131072, thinking=false. Run on Apple Silicon (Metal); SWE-rebench linux/amd64 images under emulation, so wall-clock is relative, not absolute.

Pass Rate = gold tests pass after agent's patch (real resolution). Patch Rate = non-empty diff produced.

</details>

---

🧪 Why the 2-bit is AWQ

Static KLD/PPL measures how closely a quant's logits track FP16 — it does not measure whether the model still fills correct tool-call arguments, which is what agentic use actually needs. At 2 bits the plain-imatrix build keeps decent tool selection but loses argument precision. We replay 25 held-out real tool-use sessions (disjoint from calibration) through llama-server, scoring per assistant turn whether the model picks the right tool (tool-sel) and fills the right arguments (param-acc). 3 reps, mean ± σ, same sampling as above.

Both builds are measured on the same held-out eval + FP16 baseline (static) and the same 25-session replay (agentic). Static: ↓ PPL / ↓ KLD / ↑ top_p is better. Agentic: ↑ is better.

Read the split: on the metric the static table headlines (median-KLD) imatrix looks better (1.571 vs 1.804), and it edges top_p (46.6% vs 43.9%) — yet the AWQ build is the one that actually tool-calls. AWQ activation-aware scaling wins where it counts: +54% tool-argument accuracy (0.17 → 0.26) and — just as important for agents — it collapses run-to-run variance (param-acc ±.009 vs ±.082; the imatrix build swings 0.51/0.48/0.38 on tool-sel across seeds, AWQ holds 0.50/0.50/0.48). (AWQ also nearly halves PPL, 1959 → 1040 — PPL and KLD disagree here because the fold shifts the whole distribution, lowering average surprise while nudging the very top token slightly.) The upshot: the static table alone would have picked the wrong build. All 60 gemma layers are standard attention, so AWQ scales the full network (120 groups); the per-channel scales fold into RMSNorm with no inference cost. The higher-bit builds (IQ3_M+) don't need it — at 3-4 bits the imatrix build already preserves arguments and AWQ's fold slightly perturbs general text, so those ship plain imatrix.

Harness: quant-tuner run_toolcall_reps.py. This lighter replay is the proxy used to rank the 2-bit builds; a full SWE-rebench re-run on the AWQ build is pending (the table's IQ2_M SWE-rebench figures are the prior imatrix build's).

---

🔬 How it was made

• Hybrid imatrix — activation energy E[a²] mixed with weight-column energy ‖W[:,i]‖²·E[a²] per tensor, collected over real coding/tool-use logs + wiki.test.raw via quant-tuner. Used for the IQ3_M / IQ4_XS / Q5_K_S builds.
• AWQ for the 2-bit — the IQ2_M build additionally applies activation-aware weight scaling: each input channel is rescaled by mean(|x|)^α (α grid-searched on the tool-use logs, sampled across the whole corpus) and the inverse folds into the preceding RMSNorm — mathematically identity in FP16, but it moves the channels that carry tool-argument signal onto finer quant levels. All 60 layers are standard attention, so the fold covers the full network; the imatrix is then collected on the folded weights. Net: +54% tool-argument accuracy vs plain 2-bit imatrix, at no inference cost. See § Why the 2-bit is AWQ.
• IQ2_M codebook — 2-bit E8-lattice non-uniform codes with per-tensor tier bumps (attention output, early ffn_down get more bits). llama-quantize decides the mix.
• Vision mmproj — the model's SigLIP-style vision tower (27 layers, 280 soft tokens/image) exported separately at Q8_0 with convert_hf_to_gguf.py --mmproj (visually lossless, 772 MB), so the encoder stays high-precision while the text path runs at 2 bits. No audio encoder is shipped (the source has none).
• Disjoint splits — calibration (imatrix), validation (per-tensor α gate), and eval (PPL/KLD) come from different corpora; the SWE-rebench holdout never appears in any calibration set.
• Toolchain: quant-tuner for imatrix calibration, llama.cpp @ f3e1828 for final quantization. Calibration logs mined with LogMiner.

---

🚀 Usage

Ollama

ollama run hf.co/pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ2_M

llama.cpp (GPU)

# Build with CUDA (-DGGML_CUDA=OFF for CPU/Metal)
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp/

# Run the server
./llama-server \
    --model gemma-4-31B-it-IQ2_M.gguf \
    --ctx-size 16384 --n-gpu-layers 999 --split-mode layer \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --parallel 1 --batch-size 2048 --ubatch-size 512 \
    --host 0.0.0.0 --port 1234

OpenAI-compatible API (Python)

import json, urllib.request

def ask(content, max_tokens=256):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Gemma 4 is a thinking model — disable or raise max_tokens
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request(
        "http://127.0.0.1:1234/v1/chat/completions",
        json.dumps(body).encode(),
        {"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("What is 1+1?"))

🖼️ Vision (text + image)

Gemma 4 is natively multimodal. The vision tower ships separately as mmproj-gemma-4-31B-it-Q8_0.gguf (772 MB) so you only download it if you need images. It pairs with any of the four quant files (IQ2_M / IQ3_M / IQ4_XS / Q5_K_S) — the text weights are identical; the mmproj just adds the SigLIP encoder + projector.

One-shot from the CLI (llama-mtmd-cli):

./llama-mtmd-cli \
    --model gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --image screenshot.png \
    --jinja -ngl 999 --temp 0.2 -n 256 \
    -p "Describe this image. What's in it?"

--jinja is required — Gemma 4's chat template is Jinja-based and the CLI aborts without it. --image can be repeated for multi-image prompts; URLs work too.

⚠️ Thinking + the CLI. Gemma 4 is a reasoning model. From llama-mtmd-cli, leave thinking on and give it enough budget (-n 800+) so the answer survives the reasoning preamble — the --chat-template-kwargs '{"enable_thinking":false}' flag currently returns an empty completion on the CLI path. To get a clean, reasoning-free answer, disable thinking over the HTTP server instead (below).

Vision server — host the quant with the mmproj attached (this is exactly how the worked example above was generated). --jinja is required; the vision tower is loaded via --mmproj:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --jinja --ctx-size 8192 --n-gpu-layers 999 \
    --host 127.0.0.1 --port 1234

Vision is purely additive — drop the --mmproj flag and you're back to the identical text-only model.

The OpenAI-compatible /v1/chat/completions endpoint then accepts image_url content parts. With chat_template_kwargs.enable_thinking=false the server returns just the answer (no reasoning preamble). This is the exact call used to generate the mecha prompt above:

import base64, json, urllib.request

with open("mecha.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

body = {
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": (
            "Look at this image and write a single, detailed text-to-image "
            "generation prompt that would recreate it. Cover the subject, colors, "
            "pose, lighting, style, and background. Respond with only the prompt."
        )},
        {"type": "image_url",
         "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ]}],
    "max_tokens": 400,
    "temperature": 0.3,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request(
    "http://127.0.0.1:1234/v1/chat/completions",
    json.dumps(body).encode(),
    {"Content-Type": "application/json"},
)
print(json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"])

Worked example — same image, all three quants

Each shipped quant was handed the mecha on the right and asked to "write a single, detailed text-to-image generation prompt that would recreate it".

<img src="mecha.png" alt="anime mecha — white/lavender armor, neon-green energy sword, purple railgun" width="420" /> <br clear="right"/>

⚡ Speculative decoding (MTP drafter)

This repo also bundles a multi-token-prediction (MTP) drafter at the repo root, mtp-gemma-4-31B-it.gguf (358 MB, Q4_K_M) — a self-quantized conversion of google/gemma-4-31B-it-assistant (arch gemma4-assistant, nextn_predict_layers = 4). The acceptance rates where the model's drafted tokens were accepted by the trunk were measured using three reps over 5 mixed coding/reasoning prompts × 200 tokens, temperature=0.3, thinking off; distinct seeds per rep.

Acceptance holds up across all four trunks — the highest-fidelity Q5_K_S leads at every draft depth (87.9% at n=1, still 66.0% at n=4), and even the 2-bit IQ2_M accepts 83% of single-token drafts.

How small can the drafter be? Fixing the trunk at IQ2_M and quantizing the drafter itself shows acceptance barely moves down to Q4_K_M, then erodes at 2-bit — and the erosion grows with draft depth (n):

Q6_K and Q4_K_M are statistically free — indistinguishable from the Q8_0 drafter at every depth (within ±1σ); the drafter only has to be directionally right for the trunk to accept.

Usage — add --model-draft + --spec-type draft-mtp to the server command:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --model-draft mtp-gemma-4-31B-it.gguf \
    --spec-type draft-mtp --spec-draft-n-max 4 \
    --jinja -ngl 999 -fa on \
    --host 127.0.0.1 --port 1234

The drafter lives at the repo root so --spec-type draft-mtp auto-discovers it when you load the trunk with -hf (no manual --model-draft needed): llama-server -hf pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ4_XS --spec-type draft-mtp --spec-draft-n-max 4.

---

🪪 License & attribution

• Inherits the Gemma Terms of Use from the base model.
• Base weights: google/gemma-4-31B-it.
• MTP drafter converted from google/gemma-4-31B-it-assistant (same Gemma Terms of Use).
• Calibration + quantization: Quant-Tuner with vendored llama.cpp @ f3e1828.
• Calibration logs mined with LogMiner.