granite-4.0-h-350m

Measured on device (edge-compat, granite-4.0-h-350m-fp16): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.5 tok/s · prefill 9 tok/s · TTFT 25.05 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 3.8 tok/s · prefill 8 tok/s · TTFT 30.43 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-fp16/CARD.md

Measured on device (edge-compat, granite-4.0-h-350m-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 11.5 tok/s · prefill 11 tok/s · TTFT 21.39 s (2026-09-05); Pixel 8a · LiteRT-LM 0.16.0 · CPU · decode 8.9 tok/s · prefill 8 tok/s · TTFT 28.01 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-int8/CARD.md

granite-4.0-h-350m — LiteRT-LM

ibm-granite/granite-4.0-h-350m converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 — the Mamba2 conv/SSM state buffers are bound through 0.15's generalized state binding (ExecutorMetadata); the 0.14 engine cannot bind them. Companion to litert-community/granite-4.0-h-1b.

Granite 4.0-h is IBM's hybrid architecture: 28 Mamba2 (selective-scan) blocks interleaved with 4 grouped-query attention blocks. The Mamba2 blocks carry per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV.

For GPU, pick _int8_gpu. It carries the same int8 weights as _int8, re-exported with the Mamba2 selective scan written as rank-≤4 batched matmuls and with prefer_activation_type = "fp32" declared. The two older files are unchanged. It is 45 MB larger because the rewrite pads each short prefill signature up to a full 256-token chunk with a stored constant instead of a PAD op — that is the trade that lets the graph run on a GPU at all.

Pick fp16 for reference-faithful quality (greedy outputs match the PyTorch model token-for-token on our probes), int8 for size and speed. On phones we recommend int8: the CPU runtime unpacks fp16 weights to fp32 in RAM (~3.7 GB peak on iPhone vs ~2.1 GB for int8), so fp16 is best treated as the desktop/quality variant.

Correctness

The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (8-step teacher-forced comparison, per-position max|logit diff| ≤ 3.9e-4, correlation 1.000000, top-1 identical at all positions) — both the chunked prefill path and the single-step decode path (rolled conv window + SSM state recurrence) reproduce the reference math. Both files pass an 8-question sanity gate at 8/8, matching the HF reference, and a prompt-length sweep (12–200 tokens through the engine's real prefill chunking) generates cleanly — with one engine-reuse caveat for int8 noted below.

Unlike the HF generate() default, the bundle does not prepend a BOS token — this matches Granite's official chat template, and at 350M scale it matters: a prepended <|end_of_text|> measurably degrades greedy answers on this model.

Known limitation — engine state reuse, not quantization: if one Engine object is reused across many conversations whose prompts share a growing common prefix, a reply can stop after a few tokens; on the int8 file this appears at chat-templated lengths 33–37, and on _int8_gpu at 37–41 (the folded scan changes the graph, so the band moves with it; a fresh Engine is clean at every length tested). Re-measured 2026-08-11 on litert-lm 0.15.0: a fresh Engine per conversation is clean at every tested length, and so is a reused engine that has not built up that prefix history. The cause is the runtime carrying recurrent state across conversations rather than anything in the weights (LiteRT-LM#3165); the fp16 file tolerates it, the int8 file does not. Ordinary single-conversation use is unaffected — if you sweep prompts programmatically, create a fresh Engine per prompt. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05, the internal CL that continued PR #3945). No release carries it yet (v0.19.1 does not); this note will be updated when one does.

fp16 file on the Mac GPU. The _fp16 file is CPU-only on the Mac GPU backend of litert-lm 0.18.0: the WebGPU delegate rejects DEQUANTIZE and EMBEDDING_LOOKUP (Empty quantization params) and rank-5 tensors, and no engine is created (measured 2026-10-12; the _int8_gpu file is the GPU file). Android GPU on 0.18.0: not tried; on litert-lm 0.16.0 the Galaxy S26 GPU also refused this fp16 file (device rows below).

Tool calls on litert-lm 0.18.0. Since the 2026-10-14 re-pack of the three files (metadata only: llm_model_type qwen2p5, and the chat template now renders tool results), the runtime parses the tool call and the tool result reaches the model; the earlier files returned the call as message text (<tool_call>{"name": "get_weather", "arguments": {"location": "Paris"}}</tool_call>). For _int8_gpu, the one GPU file of the three: the 2026-10-14 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous file and have not been re-run — the Mac GPU digit check passed on 0.18.0.

Usage

litert-lm run ./granite-4.0-h-350m_fp16.litertlm --prompt "What is the capital of France? Answer in one sentence."

The bundle carries the tokenizer and Granite's full chat template. Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; the KV budget for the attention layers allows up to 4096 tokens. _fp16 and _int8 are CPU files: their scan graph materialises rank-5 and rank-6 intermediates and every mobile GPU delegate refuses tensors above rank 4. _int8_gpu is the same model with those contractions rewritten as batched matmuls, and it delegates in full on Adreno and on Apple GPUs.

Performance

(The int8 Mac row was re-measured with litert-lm benchmark 0.15.0, -d 256 --runs 3, on an otherwise idle machine; the fp16 Mac row and both iPhone rows are the original cold-start single runs. Decode is dominated by the selective-scan running as generic float ops; per-parameter it is slower than a plain transformer of this size.)

Galaxy S26 — GPU backend

granite-4.0-h-350m_int8_gpu.litertlm is the GPU file. The other two are CPU files, and the reason is in the graph rather than in the runtime: transformers writes the Mamba2 SSD contractions as broadcast-multiply-reduce, which materialises rank-5 and rank-6 intermediates, and every mobile GPU delegate refuses tensors above rank 4.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. The two CPU files were gated 2026-08-24 and _int8_gpu 2026-08-27.

LiteRT-LM has no partial-GPU state. Anything short of full delegation is refused at engine creation, so a row above that says does not run is a refusal, not a slower path.

Speed on that phone, with the CPU control beside it

Both rows n=2 and within ±3% (GPU 486.5 / 484.0 prefill and 48.8 / 47.9 decode; CPU 305.6 / 296.8 and 97.6 / 95.2), 205-token prompt, same handset and binary, CPU taken immediately after the GPU runs with the phone already warm. Engine init is 8.6–9.6 s on GPU and is not counted in TTFT.

The GPU is a prefill win and a decode loss here. Use the GPU backend when you feed long prompts and read short answers; use CPU when you generate a lot of text. Both backends run the same _int8_gpu file.

The two CPU files are also slow on this phone's CPU. Same binary, prompt and protocol, 2026-09-05, two cold runs each with the phone at thermal status ≤ 1: _int8 10.6 tok/s prefill / 10.7–11.5 decode at a 2.9 GB peak; _fp16 9.0–13.7 / 9.5–11.3 at 5.4–6.5 GB. The rank-5/6 scan graph that keeps them off the GPU also falls outside XNNPACK's fast path, so on Android _int8_gpu is the file for both backends (97.6 tok/s decode on this CPU, above). The desktop and iPhone CPU figures for _int8 / _fp16 in the table above are unaffected.

Pixel 8a (Tensor G3, Mali-G715, 8 GB), same binary and protocol, 2026-09-07. _int8_gpu delegates in full on this GPU as well (38313 / 38313 ops) and answers the probe, and the order is the one seen on the S26: GPU 180 tok/s prefill / 20.4–20.7 decode with a 37 s engine init and a 4.3 GB peak; CPU 103–111 / 40.1–40.7 with a 3.1 s init and a 1.1 GB peak (two cold runs per backend, thermal status 0 before every run). The _int8 file on the same CPU decodes 7.0–8.9 tok/s (2026-09-05). On an 8 GB-class phone run _int8_gpu on the CPU; this repo's litertlm_manifest.json recommends cpu for both Android device classes on these pairs.

iPhone 17 Pro

_int8_gpu runs on both backends: 7/8 on an 8-question on-device check over Metal and 7/8 on CPU, with byte-identical greedy output between them and matching per-question verdicts (the miss is a rhyme question, not arithmetic or fact). Engine init 13.6 s on Metal, 1.0 s on CPU. That check is a different harness from the desktop one above and its questions differ, so the two scores are not comparable to each other.

Reproduce

Conversion script, litert-torch patch (Mamba2 export-cache layer + decode-state continuation + prefill pad guard), and gate harnesses: https://github.com/john-rocky/hf-to-litertlm (granite_work/).

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.