Gemma-4-E4B-it-coreml-speculative

Gemma 4 E4B IT — Core ML for Apple Silicon Macs (Speculative)

Run Google's Gemma 4 E4B IT locally on the GPU of an Apple silicon Mac with Core ML. The bundle supports text and image input, and prompt-lookup speculation verified to preserve the same output with speculation off or on. The companion app also includes a Live Camera captioning loop.

Swift runtime & app · Mac setup & model guide · When speculation helps · Verification

Who this is for

• For: Mac developers trying offline text/image chat or studying a stateful Core ML runtime with shared-KV layers and token-exact speculation.
• Not for: iPhone or Neural Engine deployment; this bundle uses GPU-resident MLState. The E2B bundle is the iPhone option.
• Not for: long-context work; this build has a 2,048-token context.

Requires: Apple silicon Mac · macOS 26+ · companion Swift runtime · roughly 8 GB free unified memory (4.0 GB measured footprint; conditions) · ~6.8 GB disk with the image encoder.

Quick start

hf download okayuji/Gemma-4-E4B-it-coreml-speculative --local-dir ./gemma-4-e4b-speculative
git clone https://github.com/oka-yuji/coreml-llm-samples
cd coreml-llm-samples
swift run -c release --package-path CoreLLMKit corellm-chat --model ../gemma-4-e4b-speculative --stats \
  --prompt "List three fruits, one per line."

This repository contains weights and configuration. The companion runtime supplies the stateful chain, shared-KV wiring and speculation loop used by both the CLI and app. For the GUI and image input, follow the Mac guide.

Speculation is opt-in: run the same command with CORELLM_MTP_PLD=1 in the environment to enable prompt-lookup speculation. Draft length defaults to 4; CORELLM_MTP_DRAFT_LEN=6 favors verbatim-heavy work. Output is byte-identical either way. On the measured workloads, speculation helps verbatim copying and is slightly slower for composed replies; see Where speculation pays.

Sample output

Apple M4 Max (128 GB) / macOS 26.5.2, GPU, greedy (temperature 0), one process per condition, corellm-chat CLI (2026-07-28). Prompt (72 tokens with chat template):

Extract every person, place, and date from this note as JSON with keys people, places, dates: Maya flew from Osaka to Reykjavik on March 3, 2026, and met Dr. Chen and Lena Novak at the Harpa concert hall on March 5, 2026.

Output — the model wrapped its JSON in a code fence; shown verbatim, identical with speculation off and on (99 tokens to EOS):

```json
{
  "people": [
    "Maya",
    "Dr. Chen",
    "Lena Novak"
  ],
  "places": [
    "Osaka",
    "Reykjavik",
    "Harpa concert hall"
  ],
  "dates": [
    "March 3, 2026",
    "March 5, 2026"
  ]
}
```

(Across 5 interleaved off/on rounds the pair-ratio median is ×0.94–0.96.)

Two honest notes on this sample. First, on this build JSON extraction is slightly below break-even — the JSON scaffolding (\``json`, braces, key names) does not appear in the prompt, so prompt-lookup has little to draft from; speculation pays on verbatim-copy spans (next section). Second, this output is byte-identical (212 bytes, same SHA-256) to what the E2B iPhone bundle produces for the same prompt — a different model size, quantization, compute unit and runtime path landing on the same JSON.

Where speculation pays

Same machine and conditions, 5 interleaved off/on rounds, per-round pair-ratio median (2026-07-28). Rows run to EOS or a 48-token cap; the JSON row is the EOS-complete run (99 tokens):

Verbatim copy is the one clear win; everything the model has to compose — even something as regular as an ID list — sits slightly below break-even, so leave speculation off for chat-style use. A cautionary tale from this bundle's own gating: an earlier internal build with a broken chat template measured ×2.37 on the enumeration row — the model was parroting the prompt, which turns any reply into a verbatim copy. A broken template can fake speculation gains; every number on this card is from the fixed, canonical-template bundle.

The verbatim-copy gate, shown in full — prompt (Japanese, "Copy this list exactly: apple, mandarin, grape, peach, pear, strawberry, melon, watermelon, kiwi, banana"):

次のリストをそのまま書き写してください。
りんご、みかん、ぶどう、もも、なし、いちご、メロン、すいか、キウイ、バナナ

Output (identical with speculation off and on, 30 tokens):

りんご、みかん、ぶどう、もも、なし、いちご、メロン、すいか、キウイ、バナナ

The CLI run above measured 32.6 tok/s off → 109.2 tok/s on (draft acceptance 0.86); across 5 interleaved rounds the pair-ratio median is ×3.49. The output ends cleanly at the end-of-turn token — 30 generated tokens, byte-identical off and on (the speculation run's --stats reports 31: it counts the end-of-turn token, which never appears in the text). Draft length 6 beats 4 on verbatim work (+11%); with mixed workloads, draft 4 is the safer default.

Key numbers

Ship bundle, Apple M4 Max (128 GB) / macOS 26.5.2, GPU, one process per condition, warmup-excluded medians (2026-07-28):

The sample, speculation table and key numbers above come from the author's 2026-07-28-e4b-manifest-fix.md: the canonical-template ship bundle, Swift benchmark harness and companion release CLI. That lab record is not public; the public Mac guide summarizes the results and provides the off/on reproduction commands. The text-only bundle is 6.5 GB; the image encoder brings the full download to ~6.8 GB.

What's inside

• 4-chunk stateful Core ML graph (gemma4, 42 layers, hidden 2560, GQA 8 query / 2 KV heads; head_dim 256 on sliding layers, 512 on full-attention layers). Chunks 0–12 and 12–24 are stateful — their KV caches live in MLState, resident on the GPU, so the host never touches KV shapes. Layers 24–41 share the KV of layers 22/23: chunk 12–24 emits those two stores as outputs and the two stateless chunks 24–36 / 36–42 consume them as inputs.
• One graph, three roles. Sequence length is a range dimension (S = 1…128): S=1 is decode, S=draft+1 is the batched speculation verify, larger S is prefill. There are no separate verify functions and no drafter model.
• Prompt-lookup speculation drafts continuations by matching the last n-gram (n ≥ 3) against the prompt, then verifies on the main graph. Rejection rolls the position back; rejected rows are overwritten and masked, so state never diverges — that is what the lossless gates check.
• int4 chunk weights (linear, per-block 32) with an int8 argmax lm_head (tanh soft-cap 30 baked in; emits token ids, not logits).
• int8 sidecars for the token embedding and the per-layer embedding (PLE) with fp32 per-row scales — 3.5 GB smaller than fp16 sidecars, and verified output-identical to them.

Input format

The runtime reads the prompt template from manifest.json — transcribed verbatim: "promptPrefix": "<|turn>user\n", "promptSuffix": "<turn|>\n<|turn>model\n" — prepends BOS, and stops on the end-of-turn token <turn|> (id 106). A single user turn is therefore:

<bos><|turn>user
{user text}<turn|>
<|turn>model

<|turn> / <turn|> are Gemma 4's actual special tokens (ids 105 / 106); the Gemma-3-era <start_of_turn> / <end_of_turn> strings are not in this vocabulary and tokenize as plain text — a mismatched template silently degrades output (it costs ~17–18 tokens per turn and leaks a literal marker string into replies). chat_template.jinja is included for HF-ecosystem tooling.

Images and Live Camera

The bundle carries vision_fp16.mlmodelc next to the language chunks. The companion runtime looks for it inside the bundle directory, next to manifest.json, and enables image input only when it is there. An image is resized to 768x768 and becomes 256 soft tokens — 256 of the 2,048 context — whatever the source resolution.

The vision tower is the same weights as the E2B bundle's: the two vision_config blocks are byte-identical, 658 of 658 tensors are bit-identical, and the towers' activations agree at maxAbsDiff 0.000e+00. Only the projection into the language model differs — 2,560 wide here against E2B's 1,536 — which is why each bundle ships its own copy and the runtime refuses a mismatched encoder instead of producing garbage from a width mismatch.

Measured on an Apple M4 Max, one process per condition:

The vision encoder runs on the GPU while the language chain keeps the full compute-unit set; the two were sharing one accelerator before that split, and separating them is where the encode went from hundreds of milliseconds to under a hundred.

This is where the stateful graph shows against the E2B bundle. On the same 10-cycle live captioning run, E4B takes 1.44 s per cycle against E2B's 2.48 s — not because it decodes faster in proportion (32.0 vs 23.5 tok/s) but because feeding the 256 image rows costs 0.36 s here against 1.21 s there. Variable-width prefill on an MLState cache absorbs the image span in one pass; a fixed-width scheduled prefill has to break it into chunks.

Files

Delete vision_fp16.mlmodelc and the bundle is a 6.5 GB text model with image input switched off.

Operational notes & troubleshooting

• Updating a bundle you already downloaded. The app has no in-place update path: open the Models screen, Delete the model, then download it again to obtain vision_fp16.mlmodelc. Incremental refresh of a downloaded bundle is not exposed in the UI.
• First run of each new sequence shape is slower. The GPU specializes kernels per shape on first use — the first differential prefill of a session can take ~2× its steady latency, and cold load is 2.9 s. Steady-state numbers in this card are warmup-excluded medians.
• Benchmark one process per condition. Core ML keeps memory mapped after a model unloads; measuring several conditions in one process contaminates the later ones.
• Core ML compile caches grow. Repeated conversions / first-loads populate ~/Library/Caches/**/com.apple.e5rt.e5bundlecache (tens of GB over time across models). Safe to delete; the next first-load re-specializes.
• The lm_head emits argmax token ids only — greedy decoding. Temperature sampling would need a logits-head variant, which this bundle does not include.

Limitations

• Context is 2048 tokens (this build). Longer-context variants exist in the lab for the 12B sibling but are not part of this bundle.
• Speculation helps only where the reply copies the prompt: verbatim copy and quoting (×3.1–3.5 measured). Anything the model composes — enumeration answered in its own format, JSON scaffolding, free-form prose — is slightly below break-even (×0.77–0.98) — leave it off for chat-style use.
• Mac GPU only. The MLState KV design does not run on the Neural Engine, and the bundle does not fit 6 GB iPhones. CPU-only execution is not supported for this graph.
• int4 fidelity: against the fp32 reference, greedy output diverges at the first generated token on one tested prompt — a benign style fork (1、2、3… vs はい、1から10まで…; fp32 top-2 margin 0.97), not degradation. On the measured verbatim / enumeration / JSON workloads, int4 output is identical to the fp16 build's. AWQ (p999) was tried and worsened fidelity (it introduced a second divergence), so raw int4 shipped.
• Greedy only (argmax head).
• No audio. There is no E4B audio encoder; transcription is available on the E2B bundle only.
• Images do not reach iPhone here. The image path is macOS-only for the same reason the rest of the bundle is: E4B does not fit an iPhone's memory budget.
• An image costs 256 of 2,048 tokens, and the pressure of repeated image turns inside that context was not measured.
• int4 moves a small number of tokens on image prompts too. Against the fp32 reference the fp16 chain shows content divergence 0 of 209 teacher-forced positions; the shipped int4 bundle shows 11 of 209. That is quantization rather than image feeding: the same bundle on image-free text shows 8 of 240, a difference that is not statistically significant, and swapping the fp32 vision tower for the Core ML one produces the identical flip set.

Verification

Everything below was gated on this exact bundle (or its direct inputs) on 2026-07-28, one process per condition, Apple M4 Max / macOS 26.5.2:

• Conversion fidelity (fp16): the fp16 chain matches HF transformers fp32 greedy for 32 tokens exactly (#mismatch = 0). The rebuilt forward was first validated against HF at fp32 (argmax 20/20, logits max |Δ| = 2.98e-4, cos_min 0.999997).
• Lossless speculation: speculation on == off, token-exact, 10/10 gate combinations on the ship bundle (4 workloads × draft length 4/6, plus two EOS-complete long runs); 16/16 on the pre-ship build including fp16.
• Runtime equivalence: the Swift runtime reproduces the python reference chain 64/64 tokens (fp16 and int4), and batched prefill matches sequential decode exactly.
• Sidecar quantization: int8 embedding/PLE sidecars produce output identical to fp16 sidecars — a 64-token greedy probe and all four workloads, token-exact.
• Cross-bundle: the JSON sample above is byte-identical (212 bytes, same SHA-256) to the E2B iPhone bundle's output for the same prompt.
• Vision tower vs the reference: reconstructed in fp32, the Core ML vision tower matches the Hugging Face fp32 tower at cosine 1.000000 on all three test images (maxAbsDiff 3.17e-05 … 5.83e-05 against reference amplitudes of ~5.4, the same order as PyTorch's own re-association noise). Its per-layer-embedding sidecar row matches 10752 of 10752 values exactly, maxAbsDiff 0.000e+00.
• Vision tower vs the E2B bundle's: vision_config byte-identical, 658 / 658 tensors bit-identical, activations maxAbsDiff 0.000e+00 — the same tower, differing only in the projection width.
• Image input, end to end (fp16 chain): teacher-forced against HF fp32 across 209 positions, content divergence 0; 4 of 6 cases match the reference greedy 40/40 exactly. A text-heavy image was transcribed verbatim across all 12 lines. The Swift vision encoder's soft tokens are byte-identical to the Python reference on all three images.
• Image feeding is block-size independent: prefilling the 256-row image span in blocks of 128, 32, 7 and 1 produces exactly the same tokens — block 7 deliberately straddles the image span — and segment-based prefill matches plain prefill 5/5 exactly.
• The image row uses per-layer-embedding row 0, not the image token's row. An earlier record of this conversion claimed row 258880; that was wrong and is corrected here. Three independent checks agree: the reference source substitutes the pad id, a reference hook gives the image position id [0] with maxAbsDiff 0.0 against row 0 (against 16.26 for row 258880), and a Swift A/B gives 209/209 exact with row 0 against 119/209 with row 258880.
• Template canonicalization: pre-release gating caught an earlier build using a broken chat template (Gemma-3-era markers, not in this vocabulary); the release manifest pins the canonical Gemma 4 turn markers, and every performance and speculation number on this card was re-measured on the fixed bundle (the conversion-fidelity gates above are template-independent and carry over; a 64/64 exact re-gate confirmed the weights were untouched).

Related work

• okayuji/Gemma-4-E2B-it-coreml-speculative — the series' first bundle: E2B on the iPhone Neural Engine, chunked multifunction graph with host-side KV, dedicated verify functions, KV save/restore.
• A 12B Core ML speculative bundle — lube8163/gemma-4-12b-coreml-iphone-practical-chat — appeared the same week as the E2B release (2026-07-25), pairing a width-4 verifier with an MTP drafter on a 12B model. Differences here, stated as fact, not evaluation: this bundle is draft-free (prompt lookup — nothing extra to load or keep resident), its off/on outputs are verified byte-identical, and it decodes at practical speed (30 tok/s class on a Mac GPU versus that card's reported ~0.2–0.5 tok/s speculative on an iPhone 14 — largely a model-size and platform difference). I have not reproduced their bundle; the comparison is taken from their published card.

Related samples

This bundle runs with the Core ML runtime in coreml-llm-samples. For separate Core AI / Foundation Models conversions and examples, see coreai-samples. That project is a related collection for comparing the frameworks; its runtime uses different model assets and does not load this Core ML bundle.

License & attribution

• Weights: derived from google/gemma-4-E4B-it, released by Google under Apache License 2.0. This repository redistributes a quantized Core ML conversion under the same license; the full license text is in LICENSE.
• Gemma is a trademark of Google LLC. This is an independent conversion and is not an official Google release.
• Conversion, runtime and verification by okayuji. The companion runtime and demo app live at github.com/oka-yuji/coreml-llm-samples.