gemma-4-E2B-GGUF

gemma-4-E2B-GGUF

GGUF-format, Q6_K quantization of Google's google/gemma-4-E2B base (non-instruction-tuned) checkpoint, for use with llama.cpp.

This is not an official Google release. "Gemma" is a trademark of Google LLC; this repository is not affiliated with, endorsed by, or sponsored by Google.

Files

With imatrix.gguf and the same llama.cpp builds (below), the Q6_K file can be rebuilt byte for byte from a bf16 conversion of google/gemma-4-E2B:

llama-quantize --imatrix imatrix.gguf gemma-4-E2B-bf16.gguf gemma-4-E2B.Q6_K.gguf Q6_K

What was changed from upstream

Converted from google/gemma-4-E2B to GGUF (bf16) with llama.cpp's convert_hf_to_gguf.py (build b11115), then quantized to Q6_K with llama-quantize, using an importance matrix from llama-imatrix (both build b10210). The importance matrix was computed from about 250 KB of English, Polish and German text: random Wikipedia articles and excerpts from public-domain novels. That text is not included here.

The last 20 layers reuse the keys and values of earlier layers, so their own key and value projections never run and receive no importance data. llama-quantize reports "did not find weights" for them and quantizes them plainly. They are unused at inference, so this does not affect output.

The audio and vision encoders are not included; this is the text model only. No other modification was made. Architecture, tokenizer, and generation config are unchanged from upstream.

Why Q6_K

Measured against the bf16 conversion on about 22,000 tokens of held-out text in 18 languages (llama-perplexity --kl-divergence):

On a text-continuation task over 8,769 positions in 18 languages, Q6_K scored the same as bf16, while Q4_K_M scored measurably lower.

License

Distributed under the Apache License 2.0 (see LICENSE in this repository), matching the license Google publishes google/gemma-4-E2B under.

Google additionally applies use restrictions to the Gemma family that are not waived by the Apache License tag and continue to apply to these weights regardless of who redistributes them:

• Gemma Terms of Use
• Gemma Prohibited Use Policy
• Gemma Intended Use Statement

By downloading or using this checkpoint you agree to comply with the Prohibited Use Policy.

See NOTICE for the upstream attribution this repository carries forward.

Usage

brew install llama.cpp

llama-cli -hf Animateus/gemma-4-E2B-GGUF:Q6_K -p "The quick brown fox" -n 32

This is a base (non-instruction-tuned) checkpoint. Prompt it with raw text to continue, not with chat-formatted instructions.