gemma-4-E4B-imatrix-awq-GGUF

Gemma-4 E4B — Calibrated GGUFs and MTP Assistants

This repository contains GGUF quantizations of base and QAT Gemma-4 E4B, calibrated on 15M tokens of pearsonkyle/llmtk-sft-corpus-v2 at 32k context. The original study compared imatrix and AWQ at IQ2_M, IQ3_M, and IQ4_XS on both models.

On 2026-10-06, 8 target GGUFs were removed using KLD > 1 in either of the two evaluation distributions (general and instruct/tools). Lower KLD indicates closer agreement with the F16 reference. The full study is preserved below; removed results are historical and their GGUFs are no longer offered on the default branch.

Available target files

The lowest measured KLD among the available files is gemma-4-E4B-it-qat-IQ4_XS-imatrix.gguf (0.057307 general, 0.048981 instruct). The retained IQ3_M files are smaller alternatives. These fidelity measurements compare each quant to its matching model reference; they do not establish task accuracy or superiority of one base model over another.

MTP speculative decoding

Google publishes matching assistants for base E4B and QAT E4B. This repository includes their Q8_0 GGUF conversions from Unsloth, approximately 94 MiB each. Use the base assistant with base targets and the QAT assistant with QAT targets.

The target and assistant are separate GGUFs (gemma4 and gemma4-assistant architectures). No target weights were changed. Select the assistant explicitly: this repository contains two target families, and filename-based draft auto-discovery does not establish the correct pairing.

With a current llama.cpp build supporting E4B MTP (support added in PR #24282), download your chosen target and its matching assistant. For example:

hf download pearsonkyle/gemma-4-E4B-imatrix-awq-GGUF \
  gemma-4-E4B-it-IQ3_M-imatrix.gguf MTP/base/mtp-gemma-4-E4B-it-Q8_0.gguf \
  --local-dir gemma-e4b

llama-server \
  -m gemma-e4b/gemma-4-E4B-it-IQ3_M-imatrix.gguf \
  --model-draft gemma-e4b/MTP/base/mtp-gemma-4-E4B-it-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ngl 99 --gpu-layers-draft 99 -c 8192 -np 1 -fa off

For a QAT target, replace the target filename with gemma-4-E4B-it-qat-IQ3_M-imatrix.gguf (or its IQ4_XS counterpart), and replace the assistant path in both commands with MTP/qat/mtp-gemma-4-E4B-it-qat-Q8_0.gguf. The example uses -fa off because the upstream E4B draft instructions report a CUDA flash-attention failure. Metal was tested here with -fa on.

Validation: both assistants loaded and actively drafted against base and QAT IQ3_M/IQ4_XS on Apple Silicon Metal with llama.cpp b9692 (f3e182816). Two short prompts per target, 96-token cap, greedy sampling, 2k context, and draft depth 3 gave acceptance of 46–53%. Seven of eight generated answers matched the corresponding baseline exactly; one QAT IQ3_M answer differed. These are smoke checks, not a quality or long-context benchmark. No consistent speed improvement was observed on this Mac; measure throughput with your workload before enabling MTP by default.

Source revisions, SHA-256 checksums, and validation counts are in MTP/validation.json. The converted drafts are copied unchanged from Unsloth base E4B and Unsloth QAT E4B.

Original benchmark study

The archived measurements are also available as original-benchmarks.csv. The original model card remains accessible at the pre-cleanup revision.

Methodology

• Calibration: 15M tokens from pearsonkyle/llmtk-sft-corpus-v2 32k split (seed-42 shuffle, 9,597 sessions, 32k context). Imatrix collected via llama-imatrix -c 32768 --parse-special; hybrid_custom variant re-weights per-tensor. AWQ α-search via proxy quantizer, imatrix collected on folded F16.
• GQA fix: gemma-4 has 4G GQA (8 Q / 2 KV heads, 42 layers, 24 KV-grouped layers). llama-imatrix only collects K/V stats for layers 0–23; a _backfill_missing_kv_layers patch copies per-channel mean of collected K/V vectors onto the 18 missing layers.
• Evaluation: KLD + perplexity + top-p agreement via llama-perplexity on pearsonkyle/broad-domain-supplement (general-domain: 30,710 tokens; instruct/tools: 30,853 tokens). Speed via llama-bench.
• Machine: Apple Silicon (MPS), llama.cpp vendored build.