Gemma-4-E2B-it-coreml-speculative

Gemma 4 E2B IT — Core ML for iPhone (Speculative)

Run Google's Gemma 4 E2B IT on an iPhone's Apple Neural Engine, or on an Apple silicon Mac, with Core ML. This bundle includes text generation, image understanding, speech transcription, and lossless prompt-lookup speculation. The companion app also provides Live Camera captions and conversation-cache save / restore.

Swift runtime & app · iPhone setup & model guide · Measured device results · Verification

Who this is for

• For: iOS developers studying a working Core ML / Neural Engine runtime, image and audio input, and speculation verified bit-exact on device.
• Not for: a ready-to-install App Store app; these weights need the companion Swift runtime.
• Not for: long-prompt prefill on a 6 GB phone. Long contexts return through KV restore; see Limitations.

Requires: iPhone on iOS 26 with Xcode 26 for the app, or Apple silicon Mac on macOS 26+ for the CLI; ~5.9 GB disk with image and audio encoders.

Quick start

For a Mac CLI check with the published runtime:

hf download okayuji/Gemma-4-E2B-it-coreml-speculative --local-dir ./gemma-4-e2b-speculative-pal6
git clone https://github.com/oka-yuji/coreml-llm-samples
cd coreml-llm-samples
swift run -c release --package-path CoreLLMKit corellm-chat --model ../gemma-4-e2b-speculative-pal6 --compute cpuOnly --stats \
  --prompt "List three fruits, one per line."

On an iPhone: open CoreMLSamples.xcodeproj at the repository root, run the CoreMLSamples-iOS scheme on your phone, download okayuji/Gemma-4-E2B-it-coreml-speculative from Models, then choose Load in Chat. The device guide covers signing, bundle copy, speculation and checkpoint controls; downloads support progress, cancel and delete.

This repository contains weights and configuration. The companion ChunkedSpeculativeChain runtime drives the multifunction graph and host-side KV cache used by the CLI and app. Image and audio input are worked examples of feeding soft tokens into that chain; the audio notes include the numerical failure found on the Neural Engine.

--compute all uses the ANE; cpuOnly checks output without its specialization cost. The first ANE run per launch specializes kernels once; see Operational notes.

Sample output

Apple M4 Max (128 GB) / macOS 26.5.2, the corellm-chat CLI, --compute cpuOnly, greedy (temperature 0), one process per condition (2026-07-28). On-device speeds are in the table below.

Prompt:

Extract every person, place, and date from this note as JSON with keys people, places, dates: Maya flew from Osaka to Reykjavik on March 3, 2026, and met Dr. Chen and Lena Novak at the Harpa concert hall on March 5, 2026.

Output — the model wrapped its JSON in a code fence; shown verbatim, identical with speculation off and on (both 212 bytes, same SHA-256):

```json
{
  "people": [
    "Maya",
    "Dr. Chen",
    "Lena Novak"
  ],
  "places": [
    "Osaka",
    "Reykjavik",
    "Harpa concert hall"
  ],
  "dates": [
    "March 3, 2026",
    "March 5, 2026"
  ]
}
```

Both runs produced byte-identical output (212 bytes, identical SHA-256) — "lossless speculation" means only the speed changes, never the text. Here speculation was worth ×1.22 (19.0 → 23.1 tok/s) because the copied spans — names, places, dates — are drafted straight from the prompt; on free-form prose it is near break-even.

Mac sample source: the author's 2026-07-28-e2b-published-bundle-mac-spec-ab.md, measured with the published pal6 bundle. It records the three matching output hashes and the off/on timings above; the underlying lab record is not public.

Verified on device

Measured on iPhone 15 (A16 / 6 GB) and iPhone 17 Pro (A19 Pro / 12 GB), iOS 26, Neural Engine, via the companion DemoApp / CLI, one process per condition unless noted. Every message logs to Documents/metrics.jsonl; the device records are dated 2026-07-25 … 2026-07-27.

The on-device lossless gate, verbatim. The byte-identical device result in the table was measured with a verbatim-repeat prompt on the iPhone 15 (A16) Neural Engine — the case where prompt-lookup speculation fires hardest.

Prompt (Japanese — "Repeat this word sequence exactly three times: apple banana orange grape peach"):

次の単語列をそのまま3回くり返して書いてください: りんご ばなな みかん ぶどう もも

Output (identical with speculation off and on, 138 bytes — the five fruit words repeated three times):

りんご ばなな みかん ぶどう ももりんご ばなな みかん ぶどう ももりんご ばなな みかん ぶどう もも

Plain decode ran at 11.9 tok/s (83 ms/tok) after a one-time 63.5 s ANE kernel specialization on that launch; speculation fired for 5 rounds and accepted 26 of 35 drafted tokens (pldAcceptRate ≈ 0.74; separately logged draftAcceptanceRate 0.775), its trajectory matching a Mac cpuOnly reference bit-for-bit.

Acceptance-metric clarification (2026-10-23): the earlier card placed 0.775 beside 26/35. The 2026-07-25 device record reports two distinct fields: pldAcceptRate 0.74 and draftAcceptanceRate 0.775. The count ratio is 26/35, approximately 0.74; 0.775 is retained as the separately logged metric, not as that fraction. The source is the author's 2026-07-25-e2b-r4a-pld-device.md (lab record, not public).

On the on-device speedup, honestly: the Mac run under Sample output shows a clean lossless ×1.22 (19.0 → 23.1 tok/s). On the 6 GB phone, though, the warm on/off speedup was never cleanly isolated — the first speculation run pays a one-time verify-graph specialization, and attempts to capture a warm on/off pair were cut short when the device degraded under memory pressure. So for the phone I report speculation as lossless with measured acceptance, not a device speedup multiplier. Acceptance is high on verbatim / quoting / repetitive / extraction spans (draftAcceptanceRate 0.775 on the device verbatim run, 0.54 on the Mac extraction) and near break-even — mostly fallback — on free-form chat; a structured JSON-extraction restore run measured 0.19. Selecting a friendly prompt would inflate the multiplier, so I don't quote a device one.

Image, audio, and Live Camera

The same bundle carries two extra encoders next to the language chunks. The companion runtime looks for them inside the bundle directory, next to manifest.json, and turns each feature on only when its encoder is present — a bundle without them still chats, with the attachment buttons simply absent.

Follow-up questions about an attached image cost nothing extra: the image stays in the KV cache, so the follow-up prefills only the new words (0.66 s to first token against 5.92 s for the turn that carried the image).

Live Camera is a demo in the companion app rather than a property of the bundle: it captions the camera feed cycle after cycle, in English or Japanese, resetting the context every cycle so a caption never drifts on the previous one. Every cycle prefills the same 279 tokens, which is how the reset is checked rather than assumed.

Measured with the companion DemoApp, one process per condition:

The first Live Camera cycle is the expensive one. Cold, it takes 80 s on an iPhone 17 Pro — ANE kernel specialization, not inference. Pressing Start therefore opens a Preparing… gate that pays about 70 s of it up front (measured 69.62 s / 69.37 s / 69.55 s across three cold runs), which leaves a 5 s first caption instead of a 80-second stare and keeps the thermal state at fair instead of serious. The compiled result is cached per device, so later launches skip it.

The audio tower is where fp16 on the Neural Engine bites. Converted naively, the fp16 audio graph computes garbage on cu=all — per-token cosine 0.53 mean, -0.045 min against the reference, while the same graph on cpuAndGPU scores 0.999998. The cause is a 1024-dim square sum reaching ~2.1e8 against an fp16 ceiling of 65504, over 3000x past overflow. Keeping one operation (softmax) in fp32 fixes it at no measured speed cost (86.3 ms vs 126 ms for the plain-fp16 graph). The int8 audio tower was converted and then dropped: it is deterministically broken on cpuAndGPU (cosine 0.127) and, even where its cosine looks fine, it flips downstream tokens at margins up to 9.7. fp16 ships for that reason, and the tower is pinned to cpuAndGPU.

What's inside

• 3-chunk multifunction Core ML graph (architecture: gemma4, 35 layers, hidden 1536; chunk bounds [0,15] [15,25] [25,35]). Each chunk is one .mlmodelc exposing a decode function and offset-prefill functions, driven by a host-side KV cache — not MLState. That host-side KV is what lets the graph run on the ANE and lets the KV be dumped to disk.
• pal6 palettized chunk weights plus int8 token-embedding and per-layer-embedding (PLE) sidecars (embed_int8.bin, ple_int8.bin, each with a *_scale_f32.bin). sidecarStage = int8.
• Prompt-lookup lossless speculation — one verify width only (prefill4 + a batched lmhead_v4 head), the memory-minimal single-width policy. The batched verify head is pal6 (grouped-16, ~291 MiB, argmax-lossless against the int8 head) and runs on cpuOnly while the forward pass stays on the ANE. Running that batched head directly on the ANE is not lossless — the batched forward's near-ties flip an argmax that plain sequential decoding does not — so the head is pinned to cpuOnly to keep output bit-exact.
• lmhead (int8, ~385 MiB) is the plain decode head; lmhead_v4 (pal6, ~291 MiB) is the batched verify head.
• KV cache save / restore — restore is decode-only (the wide prefill functions are never loaded), so a long prompt can be prefilled once and resumed later without paying prefill again.
• Bundle format id coreml-corellm-r1 (provisional; declared in manifest.json format).

Speculation & memory — read this before you judge the footprint

The metrics show an app footprint of a few hundred MB, which is easy to misread. The honest picture is three ledgers:

• App footprint (what tools show) — the app's own phys_footprint, 229–395 MB in short chat (the mem field in the metrics, and what Xcode's gauge reports).
• OS-wired ANE weights (not in the app's footprint) — on the Neural Engine the model weights are wired by the OS outside the app's accounting, on the order of 1.5–1.7 GB for E2B (my environment cannot read this directly; the range brackets an external same-class measurement of 1.74 GB and the samples guide's ~1.5 GB estimate). Wide prefill adds roughly another ~1.3 GB while resident.
• Clean mmap sidecars (not charged) — the int8 sidecars (~2.6 GiB of files) are memory-mapped; only touched rows cost real RAM.

So "a 400 MB LLM" means the app's ledger is ~400 MB while the whole device is using ~2 GB in short chat and ~3 GB+ during wide prefill. That is why a 6 GB phone can be jetsam-killed even when the app footprint looks small: the weights you asked the OS to wire count against the whole-device budget.

Low-memory default (under 7 GB RAM): speculation defaults off. On a 6 GB phone the default session is plain decode + KV restore — the configuration proven safe on 6 GB — and the verify assets are not loaded until you turn speculation on. With speculation on, a 6 GB phone can stall a few seconds per verify round when the cpuOnly head's weights are paged out under memory pressure; pal6 shrinks that head from 385 MiB to 291 MiB to reduce, not eliminate, the paging. Devices with 8 GB or more keep speculation on by default with the verify assets warmed at model load.

Prompt template

From manifest.json (promptPrefix / promptSuffix). These are not the stock Gemma <start_of_turn> markers — a loader that assumes them will silently produce a different token sequence. A single turn is BOS + promptPrefix + user text + promptSuffix:

<bos><|turn>user
{user text}<turn|>
<|turn>model

Verbatim from manifest.json (note the trailing newlines):

"promptPrefix": "<|turn>user\n",
"promptSuffix": "<turn|>\n<|turn>model\n"

BOS is token id 2 (generation_config.json); prepend it if your tokenizer does not. The runtime stops on EOS ids 1 and 106 (convert_config.json; generation_config.json additionally lists 50). Use tokenizer.json from this repository.

Files

Total download ~5.9 GB. Everything is shipped as compiled .mlmodelc folders (the loader uses them directly — no .mlpackage compile step, which on the audio tower alone would cost ~76 s on every first load).

Size units corrected on 2026-10-23: the earlier card labeled the chunk/head/embedding binary estimates as MB and the sidecar total as GB. The PLE table's old ~2.24 GB estimate was 2,240 MiB; the Hub file is 2,348,810,240 bytes. The *_scale_f32.bin companions are additional small files. Decimal MB is retained for the vision and audio encoders. This check uses the Hub inventory at revision b5d4b90ea363c8c227e66a2f80904f81a3929af0; memory measurements elsewhere retain their original units.

The two encoders are optional in the sense that the language path does not need them; delete them and the bundle is a 4.9 GB text model with the attachment buttons gone.

Requirements

• iPhone on iOS 26 (the tested target is iPhone 15 / 6 GB) with the companion Swift runtime, or an Apple Silicon Mac (macOS 26+) for the corellm-chat CLI.
• The ANE path uses computeUnits = all. The KV cache is host-side (not MLState), which is what makes ANE execution + KV dump/restore possible.
• Disk: ~5.9 GB for the full bundle (~4.9 GB if you delete the two encoders).
• Memory: see Speculation & memory. The app footprint is small (~230–395 MB) but the whole device uses ~2–3 GB; 6 GB is the tested floor, and long-prompt prefill does not fit there (use KV restore).

Operational notes & troubleshooting

• Updating a bundle you already downloaded. The app has no in-place update path: open the Models screen, Delete the model, then download it again to obtain vision_fp16.mlmodelc and audio_fp16.mlmodelc. Incremental refresh of a downloaded bundle is not exposed in the UI.
• First reply per launch is slow — once. The first inference specializes ANE kernels (tens of seconds on the phone: first-ever model load measured 120.4 s on iPhone 15, 88.9 s on iPhone 17 Pro). Later replies are fast and the cache persists on the device. If replies feel slow "every time", that is per-launch specialization, not the conversation being re-processed.
• Turning speculation on the first time takes a few seconds. On devices under 7 GB the verify assets load on the spot when you flip the switch (the specialization cost is paid then, not at model load).
• Speculation stalls a few seconds per round on a 6 GB phone. Expected under memory pressure — the cpuOnly verify head pages out between rounds. It does not corrupt output (still lossless). Devices with 8 GB+ do not show this.
• KV restore is refused ("identity mismatch"). A checkpoint carries an identity key (config hash, tensor shapes, sidecar stage, layer head-dims); restoring into a different or altered bundle is refused by design. The bundle's folder name is a record-only field and does not have to match.
• metrics.jsonl. Every message, checkpoint, and launch appends one JSON line to Documents/metrics.jsonl (token counts, finishReason, TTFT and per-token latencies, speculation stats, KV op timings, staged phys_footprint and available memory, thermal, battery). Retrieve it with the Files app (metrics.jsonl in the app's folder under On My iPhone) or devicectl device copy from.

Limitations

• Long-prompt prefill does not fit on a 6 GB phone. Feeding a ~355-token prompt SIGKILLs during prefill even at the narrowest offset width. Long context comes back via KV restore (prefill once on a Mac or a larger device, dump the KV, restore decode-only) — verified — not via on-device prefill.
• The warm on-device speculation speedup was not cleanly measured. Speculation is verified lossless on device and its acceptance rates are measured, but a controlled warm off/on speedup pair on the 6 GB phone was not captured (the device degraded under memory pressure first). The speedup you would get depends on acceptance, which is high only on verbatim / quoting / repetitive / structured spans and near break-even on free-form prose.
• Speculation defaults off under 7 GB (see above). There it carries a per-round ~3 s stall risk; pal6 reduces but does not remove it.
• Greedy decoding. The companion runtime decodes greedily (temperature 0) and the verify head emits an argmax token id — which is also what makes speculation losslessly verifiable. generation_config.json carries the base model's sampling defaults, but the shipped chain runs greedy; there is no temperature / top-k / top-p path in this configuration.
• One image or one clip per turn, and the context is still 2,048. An image costs 256 tokens and a 30-second recording up to 750, so an attachment turn plus a long conversation does not fit; the pressure of repeated image turns inside 2,048 tokens was not measured.
• Audio is truncated at 30 seconds, silently. Anything past the cap is dropped without a warning from the model.
• Japanese transcription is not post-processed and comes back with half-width spaces between morphemes.
• Image reproduction is not bit-exact from a JPEG or PNG. Feeding a decoded photo through CoreGraphics rather than the reference resampler moves the soft tokens (cosine mean 0.966 on a photograph) even though the pixel difference is negligible — a 16-layer ViT amplifies a resampler disagreement. Captions stay correct; exact reference reproduction needs the reference preprocessing path.
• The vision compute-unit split is macOS-only. Running the encoder on the GPU is the default on a Mac; on iOS the default is left alone, because whether the device GPU beats the device ANE here is unmeasured and the residency cost on a 12 GB phone is unverified.
• Mac ANE ≠ iPhone ANE. The end-to-end byte-identical on/off gate was run on the iPhone ANE (with the int8 verify head on cpuOnly). The pal6 verify head's argmax / PLD-lossless equivalence to the int8 head was gated on a Mac (cu=all); it is an argmax-identical drop-in, and a full on-device re-gate of the pal6 head is noted as follow-up in the project records.

Verification

"It runs" and "it computes the same thing as the reference" are different claims; this bundle makes the second.

• On device (iPhone 15 ANE). Plain decode and prompt-lookup speculation produced byte-identical output (138 B); the speculation trajectory (5 rounds, 26/35 accepted (pldAcceptRate ≈ 0.74; separately logged draftAcceptanceRate 0.775)) matched a Mac cpuOnly reference bit-for-bit (2026-07-25).
• KV portability (cross-machine). An int8 KV cache generated on a Mac (cpuOnly) and restored on the iPhone ANE produced a byte-identical continuation of a 355-token structured-extraction context (2026-07-26).
• pal6 verify head (Mac, cu=all). The shipped pal6 grouped-16 verify head is argmax-identical to the int8 head on real prefill hidden states (exact 4/4; margin 56/56, zero flips) and PLD-lossless on cpuOnly (exact 5/5). The head-on-ANE variant was found non-lossless for both int8 and pal6, which is why the head is pinned to cpuOnly.
• Vision encoder vs the reference. Reconstructed in fp32, the Core ML vision tower matches the Hugging Face fp32 tower at cosine 1.000000. Its per-layer-embedding sidecar row matches 8960 of 8960 values exactly, difference 0.
• Vision, end to end on a Mac. Teacher-forced against the Hugging Face fp32 reference across 221 positions: content divergence 0 (the residue is fp16 summation-order near-ties, arbitrated by margin). A text-heavy image was transcribed byte-identical to the reference. The Swift preprocessing matches the Python reference at maxAbsDiff 0.0, and the Swift encoder's soft tokens match at cosine 1.000000.
• Audio front end. The Swift log-mel is bit-identical to a float64 ground truth on all four test clips, and the numpy mel used for conversion is bit-identical to Hugging Face's own feature extractor (maxAbsDiff 0.0).
• Audio tower vs the reference. Reconstructed in fp32: cosine 1.00000000. Teacher-forced downstream argmax 1641 / 1646, with every mismatch a shared top-2 tie (reference margin max 0.166, none above 0.5).
• Transcription, end to end. Across 224 teacher-forced positions, content divergence 0; 3 of 4 clips transcribe byte-identical to the Hugging Face fp32 reference, and the fourth differs at one shared-top-2 tie where the on-device text is the closer match to the source audio. The Swift chain reproduces the Python reference chain 4 / 4 byte-identical.
• Transcription on the phone. The same clip produced a SHA-256 identical transcript on an iPhone 17 Pro and on a Mac, with identical token accounting (prompt 197 / audio rows 118 / 19 generated) — Hugging Face fp32, Mac Python, Mac Swift and iPhone ANE agreeing across four environments.
• Live Camera resets its context. Every cycle of a 10-cycle run prefilled 279 tokens, a single value, on the Mac and on the phone. Streaming the caption is free and lossless: streaming on, streaming off and the pre-streaming baseline produce a 3-way SHA match.
• Conversions are gated against the PyTorch (Hugging Face) reference with greedy top-1 matching; near-ties (fp16 summation-order differences) are arbitrated by margin, not silently accepted.

Related work

• For the Mac GPU variant in this collection, see Gemma 4 E4B IT — Core ML. This E2B bundle targets iPhone Neural Engine use; the E4B bundle targets the Mac GPU.

A 12B Core ML speculative bundle — lube8163/gemma-4-12b-coreml-iphone-practical-chat — added speculative decoding the same week (2026-07-25), pairing a width-4 verifier with an MTP drafter on a 12B model. The differences here are stated as fact, not evaluation:

• E2B-class speed. This bundle decodes at ~12 tok/s on iPhone 15 and ~16 tok/s on iPhone 17 Pro, versus that card's reported ~0.2–0.5 tok/s speculative on an iPhone 14 — largely a model-size difference (E2B vs 12B).
• Draft-free. Speculation here is prompt-lookup (no separate drafter model to load or keep resident).
• On-device receipts. This card publishes an on-device byte-identical on/off gate and a cross-machine KV-restore byte-match; and it ships the companion runtime so the verify functions actually execute.

I have not reproduced their bundle; the comparison is taken from their published card.

Reproduction

• The logs. The device figures above come from Documents/metrics.jsonl — full per-token arrays, speculation stats, KV timings, memory, and thermal state per message / checkpoint / launch.
• The lossless check is one command pair on a Mac. Run the same prompt with and without speculation and diff the output (the two produce the same text); see the CLI section of the device guide.
• On-device build, bundle copy, and speculation / checkpoint walkthrough: docs/e2b-speculative-device.md.

Related samples

This bundle runs with the Core ML runtime in coreml-llm-samples. For separate Core AI / Foundation Models conversions and examples, see coreai-samples. That project is a related collection for comparing the frameworks; its runtime uses different model assets and does not load this Core ML bundle.

License & attribution

• Base model: google/gemma-4-E2B-it. The artifacts here are derived from Gemma 4 E2B IT: the original weights have been modified — converted to a Core ML graph, quantized (pal6 palettization + int8 embedding / PLE sidecars + int8 and pal6 heads), split into three multifunction chunks, and given added offset-prefill and verify functions.
• License: Apache 2.0. Google releases Gemma 4 under the Apache License 2.0, and these derivative artifacts are redistributed under the same license. The full text is included as LICENSE.
• No source code here. This repository contains model weights and configuration only. The companion Swift runtime on GitHub is separately licensed under MIT.
• Built with Gemma.
• "Core ML" and "Apple Neural Engine" are trademarks of Apple Inc.; "Gemma" is a trademark of Google LLC. Provided as-is, without warranty.