Jan-nano

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, jan-nano): Mac Studio M4 Max · LiteRT-LM 0.14.0 · GPU · decode 69.0 tok/s · prefill 970 tok/s · TTFT 278 ms (2026-07-23); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.5 tok/s · prefill 11 tok/s · TTFT 25.51 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/jan-nano/CARD.md

Measured on device (edge-compat, jan-nano-model-block32): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 9.3 tok/s · prefill 102 tok/s · TTFT 2.08 s · all 1674 ops delegated (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 5.9 tok/s · prefill 29 tok/s · TTFT 7.14 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.4 tok/s · prefill 10 tok/s · TTFT 28.00 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/jan-nano-model-block32/CARD.md

Jan-nano — LiteRT-LM (blockwise int4)

Menlo/Jan-nano converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Jan-nano is a 4B deep-research agent fine-tuned from Qwen3-4B (Qwen3ForCausalLM) with a multi-stage RLVR recipe, optimized for tool use via the Model Context Protocol (MCP). It is a reasoning model — it emits a <think>…</think> chain before its answer — so it rides the existing Qwen3 converter and runtime directly.

⚠️ It's a reasoning model — give it room to think

Jan-nano generates a <think>…</think> reasoning chain, then the answer. Run it with max_tokens ≥ 2048 — at a short limit it gets cut off mid-thought and never reaches the answer. (All quality numbers below were measured at 2048.)

Which file?

Use model.litertlm (block 128) — for a reasoning model that emits long <think> chains, faster decode matters, and block 128 (¼ the scales → lighter GPU dequant) is ~40% faster while matching block 32 on accuracy here. It is also the build that loads reliably on iPhone (the block-32 build's larger section sits at the device memory edge). block 32 is provided for desktop/Android where the extra granularity is free.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The desktop rows are the shipped model.litertlm (block 128); the block-32 build was not re-measured, so the figures for it in “Which file?” above are the older ship-gate numbers. The iPhone row is carried over from this repository's own earlier on-device note for the block-128 build; its run log is not retained here, so the run count and prompt are not known.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 2048, identical prompt and answer-extraction for every row).

int4 is at parity (−4 pt for the recommended block-128 build). Note: evaluating a reasoning model at a short token budget badly understates int4 — at max_tokens 1024 the same block-32 build scored only 63% purely because the longer int4 reasoning chains were truncated before the answer; at 2048 it recovers to 85%. Always benchmark reasoning models with enough headroom.

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "Plan how to find where HTTP retries are configured in a Python repo."

The .litertlm bundle carries the tokenizer and prompt template (Qwen3 ChatML — <|im_start|>role\n…<|im_end|>, stop token <|im_end|>), so no separate tokenizer files are needed. The model will produce a <think>…</think> block followed by its answer.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs .litertlm models on-device:

1. Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
2. Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/
3. In the app tap +, pick the file, choose the GPU backend, and raise the max-tokens setting.
4. Chat — the bundle already carries the tokenizer and Qwen3 chat template.

A 4B int4 build needs ~2.5 GB free RAM; reboot the phone first if memory is tight.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Jan-nano model.litertlm jan-nano
litert-lm run jan-nano     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone

Verified on iPhone 17 Pro (LiteRT-LM Swift runtime): model.litertlm (block 128, 1.94 GiB section) loads and generates at ~14 tok/s. The block-32 build's section (2.11 GiB) sits at the device memory ceiling and may fail to load — prefer block 128 on iPhone.

Conversion

Converted with the official litert-torch converter — Jan-nano is a standard Qwen3ForCausalLM, so it uses the existing Qwen3 path with no custom graph code. Recipe: blockwise int4 + OCTAV (INT4 weights, block 128 or 32, symmetric, OCTAV optimal-clipping), embeddings INT8, KV cache 4096.

from litert_torch.generative.export_hf.export import export
export(
    model="Menlo/Jan-nano",
    output_dir="out",
    quantization_recipe="qwen3_int4_block128_octav.json",  # blockwise-128 int4 + OCTAV, int8 embeddings
    cache_length=4096,
    externalize_embedder=True,
)

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License

Apache-2.0, inherited from the base model Menlo/Jan-nano (itself fine-tuned from Qwen/Qwen3-4B, also Apache-2.0).