Gemma-4-E2B-it-Hi-Fi-GGUF

<img src="https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF/resolve/main/Gemma-4-E2B-it-Hi-Fi-GGUF.png" alt="Gemma-4-E2B-it-Hi-Fi-GGUF">

Gemma-4-E2B Q4_K_M GGUF, fraQtl Hi-Fi (calibration-aware quantization)

Q4_K_M at 3,461,675,328 bytes (−0.03% vs the pinned community Q4_K_M): KLD to the original bf16 model 0.05846 code/math and 0.09646 general, vs 0.09139 and 0.13249 for bartowski Q4_K_M (−36.0% / −27.2%). The 2,856,122,688-byte Phone artifact (−17.5% bytes) measures 0.06231 / 0.09716 (−31.8% / −26.7%) against the same baseline.

Method: fraQtl calibration-aware per-tensor quantization: the standard llama.cpp Q4_K machinery, with per-tensor precision decisions driven by a calibration analysis instead of fixed defaults. Every number on this page is measured on the exact files in this repo and pinned in the provenance table below.

Model Overview

Description

Two GGUF quantizations of google/gemma-4-E2B-it.

Fidelity metric: symmetric top-20 KL divergence on Q8 teacher support, mean over positions; teacher is a Q8_0 quantization of the ORIGINAL bf16 checkpoint. Lower is closer to the original model's token distributions.

On-device: iPhone, airplane mode

Unedited screenshots from our demo assistant running this model family entirely on an iPhone: on-device speech recognition, this model for understanding, deterministic Swift code executing the actions. The airplane icon is in every status bar; there is zero network at runtime.

The app says it booked; the Calendar shows it did

Scope, receipts, and speed

Dijkstra's algorithm in C++, generated offline at ~11 tok/s interactive.

Every answer carries its context tag.

Measured on an iPhone-16-class device: ≈516 MB resident · ~12 tok/s interactive (last turn, includes prefill) · 21.6 tok/s sustained on the 256-token greedy benchmark (gate ≥15: PASS) · sha256-verified artifact · llama.cpp b10075 · Metal · mmap · airplane mode throughout.

What it does (all offline): voice input and spoken replies, calendar questions, find-a-slot-and-book, move/retitle events, chained reminders, contacts-aware event titles, reading documents and photos, general knowledge, code generation.

What it does not do, by design and labeled as such in the app: no email, no messages, no network calls of any kind (there are no connectors, and it says so when asked); no action without a receipt chip (if a write fails, no chip appears); the model never announces an action itself; a deterministic code layer executes and reports.

Artifact note: the demo runs Gemma-4-E2B-it-fraQtl-SmartEdge-IQ3XXS.gguf, in this repository, 2.45 GB, sha256 761c7002…816776, byte-identical to the sha shown in the receipts panel screenshot above. Fidelity receipts (KLD ladder) for this edge variant are the next lane item; its published evidence today is the on-device behavior and benchmark shown here. The Q4_K_M and Phone artifacts carry full fidelity receipts.

License/Terms of Use

Apache 2.0 for the quantization artifacts and receipts in this repository. The model weights derive from google/gemma-4-E2B-it and remain subject to Google's Gemma terms (license link). Nothing here modifies the base model's own usage terms.

Use Case

On-device and edge inference of Gemma-4-E2B-it via llama.cpp and llama-cpp-python where fidelity to the original model matters at 4-bit-class sizes: assistants, code/math workloads, and phone/laptop deployments (the Phone artifact targets memory-constrained devices at 2.86 GB).

Release Date

Hugging Face 07/2026 via this repository.

References

• Serving lane (KV-cache compression for vLLM): Qwen3-4B kit · Mistral-7B kit
• All fraQtl artifacts: huggingface.co/fraQtl

Model Architecture

Base model: google/gemma-4-E2B-it (instruction-tuned E2B). The GGUFs were converted from the pinned bf16 checkpoint with llama.cpp's convert_hf_to_gguf.py --outtype bf16, then quantized with llama-quantize using per-tensor --tensor-type overrides selected by fraQtl's calibration-aware analysis on a code+math calibration packet. How the per-tensor decisions are made is not published; the recipe hash, the imatrix, and every artifact hash are, so the result is verifiable end to end.

Phone artifact: identical transformer quantization; the per-layer-embedding (PLE) table is additionally demoted to Q4_K. Speed-neutral by construction: bandwidth-relevant bytes (everything except the two lookup tables) are byte-identical between the two artifacts (1,188,795,532 B).

True bits-per-weight: "Q4_K_M" for E2B is 5.3–5.9 true bits/weight because the 2.35B-parameter per-layer-embedding table dominates bytes. We publish true average bits/weight over all tensors AND transformer-resident ("bandwidth-relevant") bytes. E2B decode reads ≈1.15 GB of transformer weights per token; the PLE table is a ≈7 KB/token lookup. Size comparisons on this page are total bytes; speed-relevant comparisons are bandwidth-relevant bytes.

Input / Output

• Input: text (Gemma chat format)
• Output: text
• Context length: as the base model; evaluation on this page used ctx 2048 (calibration) and ctx 512 (perplexity sanity metric).

Calibration Dataset

• Dataset: fraQtl Apex code+math packet (sha256 9bec84a28dcb0c940047e6084561a00857fd610b8a1e148cc38e27929e0a7e02), decontaminated against the eval slices.
• Data collection method: Automated (curated public code/math text).
• Properties: full packet, 212 chunks / 434,176 tokens; importance matrix computed with llama-imatrix at ctx 2048 on the BF16 substrate. The exact imatrix.dat ships in this repository (sha256 892e5f908dc43cef60be5bf02b3246bf5da8651d88d107c9fd00c20771d83e3c) so the quantization is reproducible.
• Recorded deviation from the 26B build pattern: single full-packet imatrix; no imatrix-budget sweep (E1 lean scope).

Usage

Receipt-validated stack: llama.cpp @ 78d2f524682d9fee790a6460c93d018dafeb5229 (2026-07-05) / llama-cpp-python==0.3.33 (vendors that commit). Newer releases should work; the pin is what every number on this page was measured with.

pip install llama-cpp-python==0.3.33
hf download fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf --local-dir .

from llama_cpp import Llama

llm = Llama(model_path="Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf", n_ctx=4096, seed=0)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain in two sentences why the sky is blue."}],
    temperature=0.0,
)
print(out["choices"][0]["message"]["content"])

Or with llama.cpp directly:

llama-cli -m Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf -cnv -p "You are a helpful assistant."

Chat template / turn-token note: the Gemma chat template is embedded in the GGUF; use create_chat_completion (or llama-cli -cnv) and let the runtime apply it. Gemma turns are delimited by <start_of_turn>user / <start_of_turn>model and terminated by <end_of_turn>, and the assistant role is named model, not assistant. If you hand-roll prompts, put <end_of_turn> in your stop set: stacks that only stop on <eos> run past the turn boundary and generate extra turns.

Evaluation

All evaluation is fidelity to the ORIGINAL gemma-4-E2B-it (the axis a user migrating from the bf16 model experiences), on an A100-80GB with seed 0, deterministic settings, 3 independent runs where stated (run-to-run drift 0.0). Eval slices: code/math 400-record held-out (sha256 cce6860274ab715e3ab8493c9f817b1c29ff99c148f83a25de8944e027db1665); general chat/tool/wikitext 400-record (sha256 b10a79caf2c17cc10cd1edcae44d4655278baf5b09b6a867b4d3ade2f996b276).

Fidelity to the original model (KLD, lower is closer)

QAT rows: the QAT GGUFs descend from google/gemma-4-E2B-it-qat-q4_0-unquantized, a retrained checkpoint. Unsloth's published E2B QAT mean KLD 0.00173 is measured against that retrained checkpoint. All rows above measure fidelity to the ORIGINAL gemma-4-E2B-it; the two numbers answer different questions.

GSM8K

95% band at n=200: ±7.42pp (1.96·sqrt((p₁(1−p₁)+p₂(1−p₂))/n), computed for this model); the +7.0pp difference is inside it. Protocol: 0-shot with a chain-of-thought instruction; no few-shot exemplars; not the full 1319-item set. Phone artifact: not run (lean scope).

Wikitext-2 perplexity (sanity metric)

The comparator is lower. Both absolute values are harness-inflated: ctx-512 chunked raw text on an instruction-tuned model is a sanity metric, not a capability measure. The code/math-domain imatrix is the likely cause of the relative gap (domain shift). The general-slice KLD, which includes a wikitext-anchor domain, is 0.09646 vs 0.13249 for the same two artifacts.

Scope

• GSM8K n=200 only; no MMLU, MATH-500, or long-context retrieval benchmarks were run for this model (E1 lean scope).
• Calibration domain is code+math. Both eval slices (including the general chat/tool/wikitext slice) are reported above; calibration is not domain-neutral by construction.
• Phone artifact: KLD-evaluated (single run) and smoke-tested; GSM8K and PPL were not run on it.
• Quantization is lossy by construction; the evidence standard here is measured distributional fidelity (KLD) with pinned receipts, not bit-identity.
• The base model's own limitations and biases are inherited unchanged.
• If your workload is raw web-text continuation rather than chat/code/math, weigh the Wikitext-2 row (20009.7 vs 18005.3) accordingly.

Methods note

• Deterministic eval rig: 3 independent KLD runs are byte-identical (drift 0.0). KLD is reported by mean; no confidence-interval claim is made on KLD. GSM8K is reported with the ±7.42pp band above.

Provenance

MTP speculative decoding (works with this quant)

Gemma 4 ships official MTP drafters (trained by DeepMind, distributed by Unsloth). The drafter pairs with this Hi-Fi quant out of the box: measured 74% draft acceptance at n-max=2 with byte-identical greedy output (llama.cpp b10280, llama-server, Apple Silicon; single tool-call prompt, greedy, warmup run first). Full numbers: receipts/MTP-DRAFTER-RECEIPT-2026-08-06.md.

Grab mtp-gemma-4-E2B-it.gguf (93 MB, sha256 9eba8199…191919) from unsloth/gemma-4-E2B-it-GGUF, then:

llama-server -m Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf \
  --spec-type draft-mtp --spec-draft-model mtp-gemma-4-E2B-it.gguf \
  --spec-draft-n-max 2

Desktop: on Apple Silicon, 51.0 → 52.5 tok/s in our test; at 51 tok/s the target is not slow enough for drafting to pay. Slow-decode targets such as phones are where it pays (first independent report below). n-max=2 was optimal in our tests; deeper drafting (n-max 4/6) was net negative on this hardware as acceptance decays (74% → 63% → 51%). Requires llama.cpp at or after the May 2026 MTP merge (PR #22673).

First independent result: Android (community-reported)

Reported by u/Sad-Enthusiastic on r/LLMDevs, 2026-08 (OnePlus 8T, Android ARM64, llama.cpp b9975; their measurement, not ours):

The reporter's comparison on the same device: a curated Q2_K_XL default at ~8–10 tok/s.

Tuning from the two datapoints: pick n-max by your device's decode speed. On a ~11 tok/s phone the deeper draft paid at ~55% acceptance (n-max=4); on a 51 tok/s Mac n-max=2 was optimal and deeper went negative. If you report numbers from other hardware, open a discussion; credited additions welcome.

Running it on Android (or any llama.cpp mobile build): the flags are identical; any llama.cpp-based runner at or past the MTP merge works (Termux/NDK build, llama.cpp-based Android frontends). Checklist:

1. Main model: Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf (2.86 GB, this repo), or the 2.45 GB SmartEdge-IQ3XXS build if disk is tighter.
2. Drafter: mtp-gemma-4-E2B-it.gguf from the Unsloth link above (93 MB; verify sha256 9eba8199…191919).
3. Same three flags: --spec-type draft-mtp --spec-draft-model mtp-gemma-4-E2B-it.gguf --spec-draft-n-max 2.
4. Memory: both files are mmap'd, so resident RAM stays well under the sum of file sizes (~516 MB resident for the main model in our iOS runs; the drafter adds a fraction of its 93 MB).
5. Reading your results: use llama-server and the /completion timings for tok/s and draft accept counts; llama-cli at default verbosity does not print draft stats, and a transient "failed to measure draft model memory" warning at init is known and recovers. Accept rate and tok/s from mobile hardware are welcome in a discussion.

Files

The fraQtl ladder

More from fraQtl

Weight quantization is the on-device lane. The serving lane, KV-cache compression sidecars for vLLM, holds nine concurrent ~128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Also: fraQtl/mistral-7b-instruct-v0.3-kv-sidecars. Org page: huggingface.co/fraQtl.