granite-4.0-h-1b

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

granite-4.0-h-1b — LiteRT-LM

Update 2026-08-28: metadata-only fix — removed the bundle's start_token, which silently prepended <|end_of_text|> (a token the reference chat stream never contains) to every prompt. Weights and tokenizer are unchanged: every non-metadata section is byte-identical, and only the file's sha256 changed. Scores measured on the previous file do not all carry over, because dropping that token shifts greedy decoding. The sanity gate was re-run on the file now published and moved from 8/8 to 7/8; the GSM8K and iPhone rows below were not re-run. Re-download if you fetched before this date.

Update 2026-08-13: re-converted from the same weights with the selective scan re-expressed as batched matmuls (all tensors rank ≤ 4, no giant broadcast intermediates). Two consequences, measured on the same machine and protocol: CPU decode is ~3.5× faster than the previous file (11.4 → 39.2 tok/s) and the graph now delegates fully to the GPU — verified end-to-end on macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). GPU execution requires fp32 activations, which the bundle now declares; see the honest notes below for the memory cost. If you downloaded the file before this date, re-download.

ibm-granite/granite-4.0-h-1b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (the published file passes the sanity gate on CPU and GPU on both 0.15.0 and 0.16.0; the speed rows below were measured on 0.16.0). To our knowledge this is the first Mamba2-hybrid LLM served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its selective scan on a mobile GPU.

Granite 4.0-h is IBM's hybrid architecture: Mamba2 (selective-scan) blocks interleaved with a few grouped-query attention blocks (this 1B/1.5B-class model has 36 mamba + 4 attention layers). The mamba blocks carry constant-size per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV (4096-token budget here).

Correctness

The rewritten scan is verified equivalent to the reference implementation at export time (a built-in numeric self-check runs on every export), and the float export behind this exact file matches the HF PyTorch model at the logits level (teacher-forced 8-step decode comparison: per-position max|logit diff| ≤ 1.1e-4, correlation 1.000000, top-1 and top-5 identical at every position). The published int8 file scores 7/8 on an 8-question sanity gate on both the CPU and the GPU backend (Mac, re-run on litert-lm 0.15.0 and 0.16.0 after the 2026-08-28 metadata fix — the same 7/8 on all four). The miss is 17 + 25, answered 43; the file before that fix answered 42 and scored 8/8. Removing the prepended token moves greedy decoding on close calls in both directions, and this is one of them: over 50 GSM8K questions the two files agree on 40 and split the other 10 six-to-four in favour of the older one, which at that sample size is noise. It generates correctly on a Pixel 8a with the whole graph on the OpenCL delegate (zero rejected ops, all subgraphs fully delegated). An earlier 8/8 on iPhone 17 Pro on the GPU was measured on the pre-fix file and has not been re-run.

One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). A probe asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac). The probe could not measure this on the _int8 file: on the fresh Engine the capital question came back as an empty reply in both runs (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue), so the comparison was not made. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

Tool calls on litert-lm 0.18.0. Since the 2026-10-14 re-pack (metadata only: llm_model_type qwen2p5, and the chat template now renders tool results), the runtime parses the tool call and the tool result reaches the model; the earlier file returned the call as message text (<tool_call>{"name": "get_weather", "arguments": {"location": "Paris"}}</tool_call>). The 2026-10-14 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous file and have not been re-run — the Mac GPU digit check passed on 0.18.0.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for both rows:

⚠ The LiteRT row was measured before the 2026-08-28 metadata fix and has not been re-run. See Correctness for what changing the start token does to a greedy score.

Usage

litert-lm run ./granite-4.0-h-1b_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.0-h-1b_int8.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Granite's full chat template (tool-calling and RAG-documents sections included). Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

For scale: the pre-update file measured 69 tok/s prefill / 11.4 tok/s decode on the same machine and protocol (CPU) — the rewrite is a ~3.5× CPU decode / ~3.6× CPU prefill win before the GPU enters the picture.

On device (cold start, single runs, 148-token prompt, quality-gate harness):

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 279-token prompt with generation capped at 256, 3 runs per backend):

On this low-end phone the GPU wins prefill (~1.7×) and TTFT and holds a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat within 1%) — full delegation, zero rejected ops, correct output.

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal), Pixel 8a (Arm Mali, OpenCL) and Qualcomm Adreno — the Galaxy S26 section below is that measurement.
• GPU inference runs with fp32 activations (the bundle declares prefer_activation_type = "fp32"; an fp16-activation formulation of the scan is unfinished work). That is where the GPU rows' extra memory goes — 2.53 GB vs 0.99 GB on iPhone above.
• On low-end Android the GPU does not win decode (decode is memory-bandwidth-bound; phone CPUs and GPUs share the same LPDDR, and the CPU path reads int8 weights while the fp32-activation GPU path reads expanded weights). The GPU still wins prefill and TTFT there. On Apple hardware the GPU wins across the board.
• The scan itself stays unquantized (float); int8 covers linears + embedding.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm granite_work/):

• Folded selective scan (the 2026-08-13 change): the HF chunked Mamba2 scan spells its contractions as broadcast-multiply-reduce over rank-5/6 intermediates that scale as chunks × chunk_len² × heads × d_state. The export re-expresses every contraction as a batched matmul with the chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU. The decode path likewise drops per-token expand/materialization in favor of implicit broadcasting.
• Export cache for Mamba2 layers: conv [B, conv_dim, K] + SSM recurrent [B, heads, head_dim, state] cache layers registered for Granite's layer types, so torch.export traces the model's own state contract.
• State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/SSM state consumed, so multi-chunk prefill composes), and the decode graph traces the single-step branch (conv window rolled by one).
• Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the SSM (dt forced to ~0) and the stored conv window is gathered at the last valid column. One GPU-specific trap here: reductions in the guard must keep their batch dimension (sum(-1, keepdim=True)) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
• Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
• Quantization: post-hoc dynamic int8 over linears + embedding only. Export-time conv-int8 measurably costs quality on this family, so the convs and scan stay float.
• GPU activation precision: the bundle's TOML declares prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.

2026-08-28 — start_token fix (weights unchanged)

The LiteRT-LM engine prepends the metadata start_token to every prompt, but this model's reference prompt has no leading BOS at all — the bundle was feeding an extra <|end_of_text|> the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.