gemma-3-270m-it-NPU-LiteRT

gemma-3-270m-it — NPU (SRQ) bundle, compiled on-device

A static-range-quantized (SRQ) .litertlm of google/gemma-3-270m-it that a Qualcomm device compiles for its own Hexagon NPU (HTP) at load time (JIT). The bundle contains no precompiled QNN context — one file serves any supported SoC, and no Qualcomm-generated artifact ships inside it.

Verified on: Snapdragon SM8850 (Hexagon v81), QAIRT 2.47, LiteRT-LM 0.16-line build with the NPU backend. Other SoCs and runtime versions are untested.

Run it

Your app needs ten Qualcomm/LiteRT runtime libraries in the APK and two Environment options. The full recipe — including the silent-CPU-fallback trap that makes a misconfigured run look successful — is one page:

Android NPU recipe

Know before you ship:

• Every load recompiles the model — ≈16 s on SM8850. The LLM runtime currently has no compile cache for the NPU path. This is per process start, not per prompt.
• The transformer runs fully on the NPU (888/888 prefill + 869/869 decode ops partitioned to HTP). The tiny auxiliary graphs (RoPE/mask/cache-update, embedder) run on CPU; measured cost ≈2.8% of decode time.
• The KV cache length is 896 by design, not 1024: on SM8850 the decode mask concat crossing 1 MiB makes the HTP apply the mask to the wrong rows and quality collapses (observed on SM8850, reported upstream: litert-torch #1184).

Measured performance

Single runs on Galaxy S26 (SM8850), QAIRT 2.47, LiteRT-LM 0.16-line build, 16-token prompt, 64 output tokens, per-prompt process start. The 16-token prompt makes the prefill column a floor, not a throughput claim. The two rows differ in cache length as noted, so they are indicative, not a controlled JIT-vs-AOT comparison.

Quality

On-device 45-prompt gate (SM8850, 64 max output tokens, greedy):

The uploaded file itself was additionally gated on the host interpreter: 28/45 correct (equal to fp32), same verdict on 41/45. The conversion pipeline is byte-reproducible in every weight and quantization parameter — two fresh builds differ only in a timestamp, a UUID, and the serialization order of one stop-token list (242 bytes of 456,853,286).

Reproduce

Stages 1–3 of litert-torch's experimental npu_export pipeline (tree upstream/main @ 8379afb plus the open fix PR #1178), gemma3-270m SM8850 preset, cache_length=896, 10 calibration prompts × 32 decode steps, stopping after the static-range quantize stage. Step-by-step: recipe §4.

File

---

Gemma is provided under and subject to the Gemma Terms of Use. This repository distributes a quantized conversion of google/gemma-3-270m-it; use of the model remains subject to those terms and the Gemma Prohibited Use Policy.

Known runtime issues

On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: this file carries stop sequences of <end_of_turn> and a newline with each of 13 characters in front (12 punctuation marks and a space), so the runtime holds the mark back to see whether <end_of_turn> and a newline follow, the answer ends on <end_of_turn> alone, and the held mark is never written. LiteRT-LM 0.17.0 and later write the held mark (StopTokenStreamFilter in runtime/core/tasks.cc, read in the source of 0.17.0 to 0.18.1). Not measured on this file: this follows from the runtime's source read with this file's stop list and tokenizer; the drop itself was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle whose stop sequences have the same form.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.