Gemma-4-E4B-it-GGUF

Gemma 4 E4B Instruct — GGUF (iPhone-optimized, text path)

GGUF quantizations of google/gemma-4-E4B-it, built and optimized for on-device inference on iPhone, iPad, and Apple Silicon Macs via llama.cpp or apps that wrap it (e.g. Haplo).

Built and quantized by jc-builds for the Haplo ecosystem. Original weights © Google, redistributed under the Gemma Terms of Use.

TL;DR

Gemma 4 is Google's 2026 successor to Gemma 3, using the Mix-of-Modality (MoM) pattern from Gemma 3n: a single set of weights with conditional execution paths that yield different effective parameter counts. E4B has ~4B active parameters during a forward pass and ~7B total parameters on disk. The base model is multimodal (any-to-any — text/vision/audio); this drop ships only the text path for now, since llama.cpp does not yet expose Gemma 4's vision/audio towers via standard chat APIs.

Available quantizations

Q8_0 was skipped to keep download sizes practical — for near-FP16 quality, run the upstream model directly on a Mac.

Pick Q4_K_M for general use. ~5GB is the largest single-file model that fits comfortably on a 256GB iPhone with other apps installed.

Performance on Apple Silicon

Approximate decode throughput at single-batch greedy decode, 2048-token context, text-only path. Measured with llama-cli.

Reference numbers — Gemma 4's MoM architecture has slightly higher per-token cost than a comparable 4B dense model because the conditional execution graph has more branches.

How to use

1. Haplo (iPhone / iPad / Mac)

The model appears automatically in Haplo's model browser on Kuzco-1.2.0+ builds (older builds use the wrong chat template — see "Chat template" below). The download URL for Q4_K_M is:

https://huggingface.co/jc-builds/Gemma-4-E4B-it-GGUF/resolve/main/Gemma-4-E4B-it-Q4_K_M.gguf

2. llama.cpp (CLI)

Requires llama.cpp at commit ≥ f9cd456 (May 7, 2026) for Gemma 4 inference support.

huggingface-cli download jc-builds/Gemma-4-E4B-it-GGUF Gemma-4-E4B-it-Q4_K_M.gguf --local-dir .

./llama-cli \
  -m Gemma-4-E4B-it-Q4_K_M.gguf \
  -p "Explain the difference between Gemma 3n and Gemma 4." \
  -n 512 \
  --temp 0.7 \
  --top-p 0.95

3. Ollama

cat <<'EOF' > Modelfile
FROM ./Gemma-4-E4B-it-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.95
EOF
ollama create gemma4-e4b -f Modelfile
ollama run gemma4-e4b

Chat template

Gemma 4 uses a brand new turn format that is not compatible with Gemma 3's <start_of_turn>...<end_of_turn>. The new format is:

<bos><|turn>system
{system}<turn|>
<|turn>user
{user}<turn|>
<|turn>model
{assistant}<turn|>
<|turn>model

The full template (in chat_template.jinja from the upstream model) also defines <|channel>...<channel|> for thinking/reasoning streams, <|tool>...<tool|> for tool declarations, and <|tool_call>...<tool_call|> / <|tool_response>...<tool_response|> for function calling. The basic chat path above is sufficient for most use cases.

The chat template is preserved in the GGUF metadata, so llama.cpp's --chat-template flag is not required.

If you are wrapping llama.cpp in your own app, make sure your prompt formatter has a dedicated path for Gemma 4 — Gemma 3's template will produce garbage tokens.

Sampling defaults

For chat / general output: temperature=0.7, top_p=0.95 (Google's recommendation for Gemma 4 base chat). For deterministic / structured output: temperature=0.0, top_p=1.0.

The GGUF stores the upstream defaults — most clients (llama.cpp, Haplo, Ollama) will use them automatically.

Quantization recipe

Built with llama.cpp at commit f9cd456 (May 7, 2026).

1. Downloaded google/gemma-4-E4B-it safetensors checkpoint via huggingface-cli.
2. Patched tokenizer_config.json — converted extra_special_tokens from list-of-strings (['<|video|>']) to dict ({'video_token': '<|video|>'}) to satisfy transformers 4.57.6's tokenizer loader.
3. Converted to GGUF FP16 via convert_hf_to_gguf.py --outtype f16. Only the text tower is emitted — vision and audio towers are not preserved in this drop.
4. Quantized to each target type via llama-quantize:

   llama-quantize Gemma-4-E4B-it-F16.gguf Gemma-4-E4B-it-Q4_K_M.gguf Q4_K_M
   llama-quantize Gemma-4-E4B-it-F16.gguf Gemma-4-E4B-it-Q5_K_M.gguf Q5_K_M
   

No imatrix calibration was used — the weights come from the upstream BF16 directly.

What's missing vs upstream

• Vision tower — not in this drop. Gemma 4 E4B can natively process images via <|image|> tokens; that path requires an mmproj companion file similar to SmolVLM, which is not yet wired into Haplo.
• Audio tower — not in this drop. Same reason.
• Q8_0 — skipped to keep downloads under 6GB.

If you need these, run the upstream model on a Mac via transformers directly.

Original model card

For full architecture, training, and benchmark details, see google/gemma-4-E4B-it.

License

Governed by Google's Gemma Terms of Use (linked above as the "license_link"). The Gemma license permits commercial use, modification, and redistribution, subject to the Use Restrictions in the Gemma Prohibited Use Policy. Read the full terms at https://ai.google.dev/gemma/docs/gemma_4_license before using these weights in production.

Gemma 4 by Google. Used and redistributed under the Gemma Terms of Use.

Acknowledgements

• Google for the original Gemma 4 weights and the open release.
• The llama.cpp team for the Gemma 4 converter and inference register that landed May 7, 2026 — without that, this drop would not exist.
• The Haplo ecosystem this drop is built for.