gemma-4-E2B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->

This model has no row on DeviceMark, the on-device LLM leaderboard.

<!-- gen-cards:devicemark end -->

Gemma 4 E2B (text) — Apple Core AI (.aimodel)

[!NOTE] Update 2026-07-15: gpu-pipelined-b2/ adds gemma4_e2b_qat_decode_int4lin_tbl and gemma4_e2b_qat_decode_int4lin_tbl_aotc_h18p re-exported with coreai-core 1.0.0b2, loadable on the OS 27 beta 3 toolchain (June-era b1 bundles fail to load there with a versioned-IR error). The original b1 tree is retained unchanged so existing apps and pinned catalogs keep working. The b2 decode bundle is the exact artifact measured on DeviceMark.

Gemma 4 E2B's text decoder converted to Apple's Core AI (the Core ML successor announced at WWDC26), ready to run on iOS 27 / macOS 27 — greedy 8/8 exact vs the Hugging Face reference on the iPhone GPU, the iPhone Neural Engine, and the Mac GPU. The GPU bundles embed custom fused int8/int4 Metal kernels inside the .aimodel (a Core AI feature); the ANE bundles are kernel-free and numerically hardened for fp16 NPU execution.

This repo publishes one set per platform × compute-unit: the best verified configuration — each file is the exact artifact behind the published numbers, nothing experimental — plus the gpu-pipelined/ fast path: ONE kernel-free graph that is the fastest decode on BOTH Mac and iPhone (Apple's coreai-pipelined engine + the zoo's engine patch stack).

Requires iOS 27 / macOS 27. Conversion code, knowledge base, Swift runner: coreai-model-zoo.

<!-- gen-cards:use-it begin id=gemma-4-e2b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

New to Core AI? Start with CoreAIKit 0.7.3. Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("gemma-4-e2b"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone --branch 0.7.3 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Gemma 4 E2B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model gemma-4-e2b --prompt "What can you do, offline?"

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let id = "gemma-4-e2b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "gemma-4-e2b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

When Apple's FoundationModels built-in model isn't enough, keep your session code and swap the model — one line. CoreAIKit's KitGemmaModel plugs supported chat bundles into the system LanguageModelSession. Tool calling depends on the model's dialect; guided generation also requires a compatible sequential engine. Check the linked support matrix before using either capability.

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.3) → product CoreAIKit
• Info.plist: none needed
• Entitlements: none on Mac; iPhone needs com.apple.developer.kernel.increased-memory-limit
• First run downloads the model — ~4,930 MB (Mac) / ~4,932 MB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

<!-- gen-cards:use-it end -->

Pick your platform (measured: iPhone 17 Pro / M4 Max, greedy, 8/8 exact vs HF)

| ★★★ VISION (VL): image+text → text | gpu-pipelined/gemma4_e2b_qat_vl_decode_int4linsym_tbl/ (Mac) or …_vl_decode_int4linsym/ + …_aotc_h18p/ (iPhone, provider+AOT) + gpu-pipelined/gemma4_e2b_qat_vl_vision/ + the QAT tables | 2.0 + 0.3 + 2.4 GB | 82.4 (M4 Max) · 25.5 (iPhone) — the text decoder + a 3-line image splice |

(ios-frontend/ is shared by both iPhone categories — download it once.)

Architecture is a 3-stage flow (Gemma 4's giant embedding/PLE tables stay out of the graph): frontend gather (mmap / int8 gather) → 35-layer decode core → 262k-vocab head(+argmax).

• iOS GPU core = int4 k-means fused-kernel monolith (16-entry codebook staged in threadgroup memory, packed nibble loads); the head does the 262,144-vocab matvec and argmax in-kernel (returns (value,index) partials — no logits readback).
• iOS ANE chunks = 6 fixed-shape chunks (the 35-layer monolith overflows the first-run ANE compile) with the two fp16 hardening fixes baked in: RMSNorm via the LayerNorm([x,−x]) identity (fp32-accumulating LN kernel) and Conv2d 1×1 projections (fp32 conv-engine MACs).
• macOS core = int8 k-means fused-kernel monolith (uint32-packed index loads).

★★★ Vision (Gemma 4 E2B VL) — image+text → text

The same QAT checkpoint's vision path, riding the text decoder via the zoo's static-inputs patch — the image span is causal on E2B (verified vs the fp32 HF mask dump), so positions/masks/KV need nothing new:

• gpu-pipelined/gemma4_e2b_qat_vl_vision/ — fixed-grid vision encoder, run once per image: patches [2304,768] f16 → image_embeds [256,1536] (square 768×768 = 48×48 patches = 256 soft tokens; ~100–170 ms).
• Decoder: Mac = gemma4_e2b_qat_vl_decode_int4linsym_tbl/ (tables in-graph, 95.2 prefill / 82.4 decode tok/s on M4 Max). iPhone = gemma4_e2b_qat_vl_decode_int4linsym{,_aotc_h18p}/ (provider mode — the tbl gather overflows an iOS per-encode scratch heap on the iOS 27 betas through 24A5418b (apple/coreai-models#201; not re-tested on the release OS); 41.2 / 25.5 tok/s on iPhone 17 Pro, footprint 1.96 GB) + the ios-frontend/gemma4_qat_gather_raw/ tables.
• Host contract: rewrite the prompt's 256 <image_soft_token> ids to extension ids V + slot, bind image_embeds [280,1536] as a static buffer (square fills rows 0..255); provider mode maps extension ids → the PLE pad row. Quantization is plain absmax int4 (--lin-sym) — the QAT-q4_0 grid; clipping compounds errors at long contexts.

Numerics: Mac engine ≡ python gate 24/24 token-for-token; margin-ruled exact vs the fp32 HF oracle (a flip only where the oracle's top-2 gap < 0.1). Details + conversion script: zoo/gemma4-vl.md.

Run it

Python (macOS 27): load with coreai.runtime.AIModel on the GPU delegate (SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.gpu())), drive frontend → core → head per token. Swift/device: push the set into your app sandbox (xcrun devicectl device copy to --domain-type appDataContainer). Walkthroughs + the burned-in gotchas: knowledge base · Swift runtime notes. Tokenizer: use the original google/gemma-4-E2B-it tokenizer files.

Two device gotchas (measured on iOS 27 beta 1, 24A5355q, 2026-06-10):

1. Verify each multi-GB copy completed (xcrun devicectl device info files …) before the app's first load — loading a partially-copied .aimodel poisons the on-device specialization cache for that content hash (later loads fail ENOENT even after the copy finishes).
2. Optional AOT: xcrun coreai-build compile <m>.aimodel --platform iOS --preferred-compute gpu --architecture h18p → a .aimodelc that skips the on-device compile (first load ~4× faster, decode tok/s identical to the plain .aimodel). The arch name follows the device identifier, not the marketing name: iPhone 17 Pro = iPhone18,1 → h18p (an h17p build fails to load with invalidCompiledModel).

⚠️ Known issue, still present on 26A428, affecting all Core AI LLMs (these bundles use the host-cache form that dodges it): the KV-write bug page (FB23024751 / apple/coreai-models#5).

★ GPU-pipelined fast path (zero custom kernels) — gpu-pipelined/

One decode-only S=1 LanguageBundle (input_ids [1,1] static, dynamic position/KV, embed + soft-capped head in-graph, and the 2.3 GB per-layer-embedding table as a STATIC graph input gathered in-graph by token id) rides Apple's coreai-pipelined engine: async non-blocking encode, on-GPU argmax, on-device KV growth. Measured (greedy; oracle 8/8, iPhone 24/24 token-identical to Mac-GPU): M4 Max 77.0 decode / 87.1 prefill · iPhone 17 Pro 30.3 / 38.9 — vs this repo's kernel monoliths (Mac 56.6–59, iPhone 22) with no Metal kernels at all.

Run contract (each item is load-bearing — full story + traps in the zoo's pipelined-engine page):

1. Swift stack = apple/coreai-models + the zoo's 4-patch stack (apps/*.patch, applied in order) — this bundle needs the EngineOptions.staticInputBuffers hook from coreai-pipelined-static-inputs.patch.
2. Bind the two table files (download from ios-frontend/gemma4_gather_raw/) as static inputs: ple_table ← embed_per_layer.i8, ple_scale ← embed_per_layer.scale.f32 — as OWNED storageModeShared MTLBuffers (read the file in once). A PROT_READ-only mmap under makeBuffer(bytesNoCopy:) silently costs ~65 ms/GB per encode on macOS; a writable COW mmap is fine on the Mac but pays a residency tax on iPhone.
3. COREAI_CHUNK_THRESHOLD=1 before engine creation (prefill = pipelined S=1 steps); never call engine.warmup() (it warms shape 256; the S=1 graph rejects it) — a 1-token generate after load is the warmup.
4. iPhone: AOT first — xcrun coreai-build compile <bundle>.aimodel --platform iOS --preferred-compute gpu --architecture h18p --expect-frequent-reshapes, then point metadata.json's assets.main at the .aimodelc (on the OS 27 betas the plain bundle passed on-device specialization but the spec'd artifact asserts at first execute) — or download the precompiled gpu-pipelined/gemma4_e2b_decode_int4lin_tbl_aotc_h18p/ (iPhone 17 Pro class). Ship the com.apple.developer.kernel.increased-memory-limit entitlement (the owned 2.35 GB table; measured peak footprint 4.4 GB vs a ~6.4 GB entitled limit) and bench a settled device (a just-unlocked iPhone under-reads ~35%).

In-app: the zoo's CoreAIChat ships this config as the Gemma ⚡ engine mode (GPU/ANE/⚡ segment) — it downloads the _aotc_h18p bundle plus the two table files and binds them as owned static buffers. Chat-surface on a settled iPhone 17 Pro: decode 32.7 / prefill 44.2 tok/s on a 200-token turn (vs 22 for the kernel-monolith GPU mode). First in-container load pays a one-time ~2 GB spec-cache ingest (~11 s engine load, ~6 s warm) and can invalidate sibling models' cached specializations once — the app's GEMMA_CLEAR_SPEC_CACHE=1 hook recovers.

The per-token-provider variant (PLE rows filled per step by a host callback — iPhone 26.5 decode / 40.5 prefill, no entitlement, clean mmap) is the lighter alternative; reproduce it from the same conversion script (conversion/export_gemma4_decode_pipelined.py, drop --tbl).

★★ Official QAT weights — int4 quality guaranteed by design

gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl/ (+ the _aotc_h18p/ precompile) is the same graph re-exported from Google's official QAT release google/gemma-4-E2B-it-qat-q4_0-unquantized: bf16 weights trained for q4_0 rounding, and q4_0 is this bundle's quantization (per-block-32 absmax-class linear int4). Google publishes these checkpoints as "preserving similar quality to bfloat16", explicitly for custom downstream compilation — so the int4 claim here upgrades from "PTQ that gates 8/8" to int4 ≈ bf16 by design. Measured: same speed as the PTQ bundle (M4 Max 78.9 decode / 89.6 prefill; iPhone 17 Pro 30.7 / 36.7 settled; oracle 8/8 on python, engine, and device).

⚠️ Pair QAT bundles with the QAT tables: bind ios-frontend/gemma4_qat_gather_raw/{embed_per_layer.i8, embed_per_layer.scale.f32} — the PLE table is checkpoint-derived, so the original gemma4_gather_raw/ files do NOT match these weights. Everything else (patch stack, chunk threshold, entitlement, AOT) is identical to the PTQ run contract above. Gemma 4 E4B (the bigger sibling, also from official QAT weights) lives in its own repo: gemma-4-E4B-CoreAI.

Parity

All three sets reproduce the HF eager greedy reference 8/8 top-1 exact ("What is the capital of France?" → "The capital of France is Paris."), verified on macOS conversion and re-verified end-to-end on device per compute unit.

License

Gemma 4 is released under the Apache License 2.0 — see the base card google/gemma-4-E2B-it and Google's Gemma 4 license page. These .aimodel bundles redistribute the weights in a different serialization; the conversion adds no additional restrictions.

CoreML (iOS 18+) variants: gemma-4-E2B-coreml · gemma-4-E2B-stateful-coreml.

<!-- funnel:v1 -->

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->