gemma-4-E4B-it-qat-GGUF

TheStageAI/gemma-4-E4B-it-qat-GGUF

A portable GGUF release of Google's Gemma 4 E4B instruction model, compressed from Google's QAT-trained BF16 weights and emitted as standard llama.cpp-compatible .gguf files.

• Run it with: llama.cpp or other GGUF-compatible runtimes
• Compression source: google/gemma-4-E4B-it-qat-q4_0-unquantized
• BF16 reference: google/gemma-4-E4B-it
• Smaller native release: TheStageAI/gemma-4-E4B-it-qat

Use this repo when deployment portability matters most. If you can run our native MLX runtime and want the smallest artifacts, use the edge-lm sibling release.

Why this exists

The native edge-lm checkpoints use custom codecs for both decoder weights and PLE tables, which is why they are smaller at comparable quality. Many deployments, however, need standard GGUF files that work with llama.cpp-compatible tooling.

This repo keeps the production bit-width schedules from our native compression pipeline, but maps the weights into GGUF-compatible quantization formats. The result is larger than the native release, but portable.

How it was compressed

We start from Google's QAT-trained BF16 checkpoint and reuse the production m and l schedules from the native release.

• Transformer blocks - the M and L files follow our RCO-selected production bit-width schedules, then emit the weights in GGUF-compatible K-quant layouts with the required group sizes and symmetric/asymmetric modes for each tensor family.
• PLE tables - stored with GGUF-compatible Q4 scalar quantization instead of the native AQLM PLE codec, so the files stay portable across GGUF runtimes.
• Token embeddings / LM head - quantized through the same GGUF-compatible path as the rest of the model.
• W4-uniform - a conservative uniform 4-bit GGUF variant with the same Q4 PLE path.

Operating points

Usage

Use a recent upstream llama.cpp build. Example:

llama-completion \
  -m gemma-4-E4B-it-qat-GGUF-L.gguf \
  -p "Explain gravity in one sentence." \
  -n 64

Benchmarks

For quality evaluation, GGUF checkpoints are converted through the same dequantized BF16 evaluation path used for the native release, so the backend is equalized. IFEval p/i means prompt strict / instruction strict, using the corrected public recipe with max_gen_toks=1280.

MMLU-Pro is the official checkpoint-wise vLLM route with Gemma chat formatting and thinking enabled. The .gguf files in this repo also passed generation smoke tests with upstream llama.cpp.

Files

License

Released under the MIT License. As a derivative of Gemma, the weights are also subject to the Gemma Terms of Use.

Citation

If you use these checkpoints, please cite the Gemma 4 release and the methods we build on (GPTQ, QEP, AQLM, RCO) - see the references in the edge-lm write-up.