granite-4.1-3b

Measured on device (edge-compat, granite-4.1-3b-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 15.3 tok/s · prefill 243 tok/s · TTFT 900 ms · all 1784 ops delegated (2026-08-24); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 7.1 tok/s · prefill 20 tok/s · TTFT 990 ms · all 1784 ops delegated (2026-08-17); Mac M4 Max · LiteRT-LM 0.16.0 · GPU · decode 38.1 tok/s · prefill 715 tok/s · TTFT 405 ms (2026-08-17). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.1-3b-int4/CARD.md

Measured on device (edge-compat, granite-4.1-3b-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 9.5 tok/s · prefill 162 tok/s · TTFT 1.35 s · all 1784 ops delegated (2026-08-25); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 3.9 tok/s · prefill 15 tok/s · TTFT 1.36 s · all 1784 ops delegated (2026-08-17); Mac M4 Max · LiteRT-LM 0.16.0 · GPU · decode 32.7 tok/s · prefill 743 tok/s · TTFT 399 ms (2026-08-17). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.1-3b-int8/CARD.md

granite-4.1-3b — LiteRT-LM

ibm-granite/granite-4.1-3b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

Granite 4.1-3B is IBM's 3.4B-parameter dense instruct model (40 layers, hidden 2560, GQA 40:8, tied embeddings, 100k vocab), long-context and tool-calling capable, released April 2026.

The int4 file is the one to use on a phone: on Apple hardware it is both smaller and faster than int8 (see below). On Android the int8 file is not out of reach either — it runs fully delegated on a Galaxy S26 GPU at a 1476 MB peak against int4's 1394 MB (see below).

Correctness

The int4 file scores 8/8 on an 8-question sanity gate on both CPU and GPU (Mac, litert-lm 0.16.0), 8/8 on iPhone 17 Pro on Metal (and 7/8 on its CPU — the one miss is the rhyme line in a single combined prompt, which the model answers correctly when asked on its own), and generates correctly on a Pixel 8a with every node on the OpenCL delegate — 1784/1784 in the 1024-token prefill graph and 1614/1614 in decode, zero rejected ops. The int8 file scores 8/8 on all the same Mac lanes and also runs fully delegated on the Pixel.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for every row:

int4 costs 4 points on this model — an ordinary blockwise-int4 result, and worth the 1.7× size reduction for a phone.

Usage

litert-lm run ./granite-4.1-3b_int4.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.1-3b_int4.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Granite's role markers (<|start_of_role|>…<|end_of_role|>), with a 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1). Six rather than eleven is deliberate: every exported signature is charged engine memory whether or not it is called, and an eleven-signature build of these same weights was killed by iOS during Metal initialisation unless the app capped the context at 1024. This file creates its engine at the full 4096 on an iPhone 17 Pro.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --cache no, quiet machine, serialized:

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Pixel 8a (Tensor G3, Mali-G715, 7.75 GB RAM; litert_lm_main built from the v0.16.0 tag, single runs):

Honest notes:

• On this low-end phone the GPU wins prefill (3.7×) and TTFT (3.3×), not decode (7.07 vs 6.98 tok/s). Decode is memory-bandwidth-bound and a phone's CPU and GPU share the same LPDDR. On Apple hardware the GPU wins across the board.
• int4 beats int8 on the GPU on both platforms — 1.2× decode on Metal, 1.8× on Mali — while int8 wins CPU prefill (2.5× on Mac), where XNNPACK's large matmuls outrun the blockwise dequant.
• A 3.4B model on a 2020s mid-range phone is not a fast experience in absolute terms; the numbers above are what the hardware gives.

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24 and 2026-08-25.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_1024; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0, from a pristine released stack — no patched checkout (reproduction script: hf-to-litertlm granite41_work/).

• No start token in the metadata. Granite's tokenizer sets add_bos_token: False, and its BOS is its EOS (<|end_of_text|>). The converter writes a start_token from tokenizer.bos_token regardless, and the runtime prepends it on the first turn — which this model reads as a document that already ended, answering by echoing the question back. Removing the field is the whole fix, and the difference is 5/8 vs 8/8 on the sanity gate. The same behaviour reproduces on the bf16 PyTorch model when the token is prepended by hand, so it is a prompt-construction issue rather than anything about the conversion or the quantization.
• Prefill ladder trimmed to six signatures — in the int4 file. Every exported signature is charged engine memory whether or not it is called. With the full eleven, engine creation on iPhone 17 Pro was killed by the OS during Metal initialisation; six (1024, 256, 64, 16, 4, 1) leaves comfortable headroom and costs only padded chunks at intermediate prompt lengths. The int8 file still carries the full eleven-signature ladder — that is the difference behind its twelve GPU subgraphs against int4's seven in the Galaxy S26 table above — so int4 is the file to use on iOS.
• Embedder externalised so the tied 100352×2560 vocab table sits in its own section, clear of the ~2 GiB single-section mmap ceiling on iOS.
• Quantization: int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding; int8 is dynamic per-channel on linears and embedding.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.