granite-4.2-3b

Measured on device (edge-compat, granite-4.2-3b-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 15.6 tok/s · prefill 235 tok/s · TTFT 940 ms · all 1783 ops delegated (2026-08-31); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 85.5 tok/s · prefill 1240 tok/s · TTFT 237 ms (2026-08-31); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.8 tok/s · prefill 22 tok/s · TTFT 9.28 s (2026-08-31). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.2-3b-int4/CARD.md

Measured on device (edge-compat, granite-4.2-3b-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 9.2 tok/s · prefill 241 tok/s · TTFT 960 ms · all 1783 ops delegated (2026-08-31); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 70.7 tok/s · prefill 1208 tok/s · TTFT 244 ms (2026-08-31); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 7.0 tok/s · prefill 51 tok/s · TTFT 4.20 s (2026-08-31). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.2-3b-int8/CARD.md

granite-4.2-3b — LiteRT-LM

ibm-granite/granite-4.2-3b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

Granite 4.2-3B is IBM's compact reasoning model (3.66B dense, 40 layers, hidden 2560, GQA 40:8, untied embeddings, 100k vocab), released August 2026. It works problems inside <think>…</think> before answering, and these bundles carry that machinery: the generation prompt pre-fills the think opener the way IBM's own chat template does, and a thought channel is declared in the bundle metadata so the runtime separates reasoning from the answer and honors a thinking budget.

The int4 file is the one that fits a phone (smaller, and faster on every GPU we measured); int8 is the quality build — it is within 1 GSM8K point of bf16, where int4 costs 11 (see Accuracy below). Pick by what your deployment can hold.

Correctness

Both files score 8/8 on an 8-question sanity gate on both CPU and GPU (Apple M4 Max, litert-lm 0.16.0 runtime lineage), with every answer arriving as a clean final answer through the thought channel — no reasoning leakage, no degeneration. The bf16 PyTorch reference scores 8/8 on the same questions under the same template.

On a Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno) both files generate correctly on GPU and CPU, with full OpenCL delegation — 1783/1783 nodes on every prefill signature and decode, zero rejected ops — and the runtime separates the reasoning on-device exactly as intended: the answer arrives as a labeled thought block followed by the clean final answer.

On an iPhone 17 Pro the int4 file passes the same gate on both backends (on-device byte count verified against the source file): Metal GPU 7/8 (engine created at the bundle's full 4096-token budget, init 13.0 s; the one miss is the rhyme line at the end of the composite 8-question prompt — the model answers that question correctly when asked on its own, on every other platform) and CPU 8/8 (init 5.5 s). Available process memory never dropped below ~4.2 GB during either leg.

Accuracy — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max_tokens 2048 (a reasoning model truncated mid-thought never reaches its answer), n=100, identical prompt, harness and answer extraction for every row; the LiteRT rows ran on the litert-lm 0.16.1 engine (CPU backend), scored on the text after the thought channel:

int8 is at parity. int4 costs 11 points on this model — noticeably more than the −4 the same recipe cost the non-thinking granite-4.1-3b of the identical shape. The obvious escape was measured and closed: a block-128 build of the same weights scores 81.0% on the same protocol (a wash), so the cost is a property of int4 on this model, not of the block size. If your deployment can host the 3.76 GB int8 file, it is the better model; the int4 file is the one that fits a phone, and this table is the price.

Usage

litert-lm run ./granite-4.2-3b_int4.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.2-3b_int4.litertlm --backend gpu --cache no --prompt "..."

Notes for a reasoning model:

• Give it a generous output budget (≥ 2048 tokens). The model thinks before it answers; truncated mid-thought it produces no final answer at all.
• The bundle declares the thought channel (<think>…</think>), so runtimes that expose ThinkingConfig / a thinking budget can cap or read the reasoning separately; the streamed answer contains only the final response.
• The bundle uses ChatML role markers (<|im_start|>…<|im_end|>) with a 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1) — six rather than eleven because every exported signature is charged engine memory whether or not it is called, a lesson measured on this model family's 4.1 ship, where an eleven-signature build was killed by iOS during Metal engine initialisation.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, serialized, ≥300 s rest before each GPU reading:

Two patterns worth knowing, both consistent with the granite-4.1-3b conversion of the same shape: int4 beats int8 on the GPU (+21% decode on top of the 1.7× size reduction), while int8 wins CPU prefill (2.5×) — XNNPACK's large matmuls outrun the blockwise dequant.

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):

The int4 GPU decode range's low end is the second back-to-back run (thermal); its high end is the rested reading. The shape matches the other platforms: int4 wins GPU decode 1.7× over int8, GPU wins TTFT ~5–10× over CPU, and peak process RSS stayed at 1.4 GB (int4 GPU) / 4.3 GB (int8 CPU) with no OOM on the 12 GB device.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1, from a pristine released stack — no patched checkout (reproduction script: hf-to-litertlm).

• The think opener is pre-filled by the bundle's prompt template. IBM's chat template ends the generation prompt with <|im_start|>assistant\n<think>\n, and the bundle's assistant prefix reproduces that exactly. This matters at export time: the converter derives the assistant prefix from an assistant history message, so a template that only opens <think> in its generation branch silently ships without the scaffold — the reasoning discipline is then left to the quantized model's own choice.
• A thought channel is declared in the metadata (<think> / </think>). Without it the runtime streams raw reasoning into the answer and silently ignores any thinking budget. The exporter does not emit it on this path; it was added post-export with the weights verified byte-identical.
• A Jinja chat template is bundled (files uploaded 2026-09-15). It is the canonical models/granite4_2 form: ChatML roles with the always-present system turn, IBM's <tool_call>/<function=…> XML for tool calls, enable_thinking (default true) choosing between the pre-opened <think> and the closed <think></think> generation prompt, and past assistant turns rendered as <think></think> + answer, as IBM's template renders history. The files before this date carried only the structured prefix/suffix; on LiteRT-LM 0.16 and 0.17 their re-rendered history dropped the assistant turns, which is why long conversations drifted (LiteRT-LM #3445). Weights and tokenizer are byte-identical to the earlier files; only the metadata section changed.
• No start token in the metadata. granite-4.2's tokenizer declares <s> as BOS but never prepends it (the tokenizer's post-processor adds nothing), so the converter's unconditional start_token write was suppressed. The 4.1 lesson, different mechanism: a prepended token the model never saw in that position degrades answers.
• Embedder externalised so the 100352×2560 input-embedding table sits in its own section, clear of the ~2 GiB single-section mmap ceiling on iOS. (4.2's embeddings are untied; the lm_head stays in the main graph.)
• Tokenizer embedded as the upstream tokenizer.json (HF tokenizer section), not a SentencePiece conversion — byte-level BPE survives intact.
• Quantization: int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding; int8 is dynamic per-channel on linears and embedding.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle (thinking pre-fill and thought-channel metadata as described). This repository is a community conversion and is not affiliated with IBM.