granite-4.0-h-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta, 2026-06).

Mirror of mlboydaisuke/granite-4.0-h-CoreAI — the canonical repo (CoreAI Model Zoo). Updates land there first.

Granite 4.0-H 1B / 350M — Apple Core AI (.aimodel)

IBM Granite 4.0-H 1B / 350M (hybrid Mamba-2 / Transformer) running on Apple Core AI — measured 35 tok/s decode on iPhone 17 Pro and 136 tok/s on an M4 Max GPU (1B int8), 191 tok/s for 350M fp16.

IBM Granite 4.0-H (Mamba2 + attention hybrid; 1b: 36 Mamba2 mixers + 4 NoPE GQA attention layers) converted to Apple Core AI for iOS 27 / macOS 27 (beta), riding Apple's coreai-pipelined GPU engine via the decode-only loop-free export — async encode, on-GPU argmax sampling, on-device KV growth, zero custom kernels.

The first SSM-scan architecture on this path: at S=1 the Mamba2 selective scan is a single recurrence step (no while_loop in the graph), and the conv/SSM states ride as two fixed-shape extra states — the same shape-class as Qwen3.5's GDN, so no engine changes beyond the existing patch stack.

Numerics: 16/16 teacher-forced single-step top-1 vs the fp32 HF oracle + HF-cache-seeded decode step (the zoo ship gate, on an oracle whose top-2 margin is ≥ 0.1 at every position), and the iPhone greedy sequences are 24/24 token-identical to the Mac GPU on both fixed prompts, both runs.

<!-- gen-cards:use-it begin id=granite-4.0-h-1b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Granite 4.0-H 1B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model granite-4.0-h-1b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "granite-4.0-h-1b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed
• First run downloads the model — 1.7 GB (Mac) / 1.9 GB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

<!-- gen-cards:use-it end -->

Bundles

• gpu-pipelined/granite_4_0_h_1b_decode_int8hu_block32_sym/ — int8lin + the tied lm_head untied and quantized absmax per-block-32 int8 (symmetric, no clipping — clipping corrupts big-vocab heads), 1.79 GB. The device ship: +17–21% decode on iPhone (the head was ~10% of the per-token read on the bandwidth-saturated surface; on the Mac it is ~flat, the engine pipeline hides the head there). Oracle gate 16/16 + decode step; device numerics 24/24 ≡ Mac-GPU on all 3 runs. (Do not re-quantize heads per-channel: per-channel axis-0 int8 is broken on the current beta GPU delegate — garbage logits.)
• gpu-pipelined/granite_4_0_h_1b_decode_int8lin/ — full LanguageBundle (metadata.json + tokenizer/ + .aimodel), int8 linear per-block-32 (scale-multiply dequant, no LUT), fp16 embed + tied lm_head in-graph, 1.63 GB. The Mac ship configuration (136.5 vs 134.2) and the lighter iPhone alternative.
• gpu-pipelined/granite_4_0_h_350m_decode_fp16/ — the 350m as fp16, 0.66 GB. At this size the model is overhead-bound, not bandwidth-bound: int8 measured slower than fp16 (185.8 vs 191.1) and fails the oracle gate (shared_mlp.output_linear is block-32-sensitive), so the 350m ships fp16.

input_ids is STATIC [1,1] (the selective scan at S=1 ≡ one loop-free recurrence step); position_ids + KV seq stay dynamic, so EngineFactory classifies the bundle dynamic → pipelined engine. States: growing KV (4 attention layers) + conv columns [36,1,conv_dim,3]

• SSM state [36,1,48,64,128] (fixed shape, carried by the extra-states patch).

Run

Needs the engine patch stack from the zoo (apps/coreai-shared-product.patch → apps/coreai-pipelined-extra-states.patch; Apple's repo is issues-only, so capabilities ship as patches), then:

COREAI_CHUNK_THRESHOLD=1 llm-benchmark --model granite_4_0_h_1b_decode_int8lin -p 128 -g 256 -n 3

• COREAI_CHUNK_THRESHOLD=1 before engine creation — prefill runs as pipelined S=1 steps (prompt tok/s ≈ decode tok/s).
• Never call engine.warmup() — it warms query length 256 and the static [1,1] graph rejects it. A 1-token generate after load is the warmup (llm-runner needs --warmup exact --warmup-length 1).
• Benchmark Release builds only (a Debug engine measures ~3× slow).

iPhone

The 1b int8lin runs at ~31 tok/s ≈ ~84% of the naive bandwidth ceiling (~60 GB/s ÷ 1.6 GB/token ≈ 37) on an iPhone 17 Pro — effectively memory-bandwidth saturated; the SSM scan costs nothing extra at S=1. Cold GPU specialization 5.7 s, warm loads 1.9 s (content-keyed cache — no AOT compile needed, and at 1.6 GB no increased-memory entitlement is required, unlike 2B-class bundles).

Reproduce

Conversion script (self-contained) + method page in the zoo: conversion/export_granite4h_decode_pipelined.py (int8lin, or fp16 --hf-id ibm-granite/granite-4.0-h-350m) · zoo/granite-4.0-h.md (includes the 350m int8 post-mortem and the oracle-margin rule) · knowledge/pipelined-engine.md

License

Model weights: Apache-2.0 (IBM Granite; LICENSE included). Conversion code: BSD-3-Clause (see the zoo).