Gemma3-1B-IT-Tensor-NPU

Gemma3-1B-IT — Google Tensor NPU (AOT, 4096 context)

[!NOTE] Quantized, repackaged variants of google/gemma-3-1b-it precompiled (AOT) for the Google Tensor Edge TPU NPU across Pixel generations. Packaged as .litertlm bundles for LiteRT-LM. G3/G4 = int4 per-channel + Hadamard · G5/G6 = int8. (This repo was previously named Gemma3-1B-IT-Tensor-G4-NPU.)

[!IMPORTANT] Community build — not an official Google release. Built and maintained by @xThr45hx; not affiliated with or endorsed by Google.

Variants

[!NOTE] Why int8 on G5/G6? On a retail Tensor G6 (santafe_a0) every int4 variant we compiled with the Tensor SDK v2.1 runs on the NPU but produces word salad. That covers int4 + Hadamard, plain per-channel int4, mixed int8-head + int4-body, blockwise-64 int4, and several compiler-option combinations (8 builds in total). The same graph quantized to int8 is coherent on the NPU. The int4 path on G6 is not yet working with the current compiler, and that is not a quant-recipe problem we could fix on our side. G5 uses the same int8 recipe as G6. Google's own G5/G6 Gemma 3 1B bundles are also int8.

The earlier int4 G5/G6 bundles in this repo have been removed. The G6 one failed warmup on a retail Pixel 11 (it was built with the older beta SDK, whose G6 firmware ABI didn't match retail silicon; SDK v2.1 fixed that). The G5 one was never confirmed on hardware.

[!WARNING] Tensor G5 is not yet confirmed on a Pixel 10. It is built from the same int8 model as the verified G6 bundle and compiles fully to the NPU. If you have a Pixel 10, please test it and comment on this repo (Community tab) with: device, whether it loads on the NPU, tokens/s, and a sample multi-sentence reply. That's what moves it to ✅.

[!WARNING] Tensor G3 is not yet confirmed on a Pixel 8. Strong evidence it works: the G3 and G4 compiles are the same program. They differ in only 19 bytes (a device-type tag and its hash), and the G3 bundle runs on a Pixel 9 (G4) and gives word-for-word identical replies to the G4 build. If you have a Pixel 8 / 8 Pro / 8a, please test it and comment on this repo (Community tab) with: device, whether it loads on the NPU, tokens/s, and a sample multi-sentence reply. That's what moves it to ✅.

Cross-generation DGC compatibility (tested on a Pixel 9 Pro XL / Tensor G4)

• G3 ↔ G4: the G3 and G4 builds compile to the same program. They differ only in a device-type field (03 vs 04) in the embedded compiler options, plus its 8-byte hash. The G3 bundle loads and gives word-for-word identical replies to the G4 build on G4 hardware, so the G4 runtime does not enforce the tag.
• G5 → G4: in an earlier test, a G5-compiled Gemma4-E2B loaded on G4 and its prefill executed on the G4 NPU, but decode did not. G5 compiles use dual-cluster optimizations that G4 hardware lacks, so newer-generation DGCs are only partly forward-loadable.
• Takeaway: use the build for your chip. G3 and G4 share a program; G5 and G6 need their own compiles.

All variants: 26-layer Gemma 3 1B-IT, 4096 context (embedded KV cache), prefill 128, embedded tokenizer + chat template, .litertlm v1, built from Google's QAT checkpoint google/gemma-3-1b-it-qat-int4-unquantized. Sizes: G3/G4 int4 ≈ 875 MB · G5 int8 ≈ 1.65 GB · G6 int8 ≈ 2.11 GB.

• G3/G4: to our knowledge, the first publicly available int4 AOT Tensor-NPU build of Gemma 3 1B, at about half the size of the int8 Tensor-NPU Gemma 3 1B bundles: 875 MB vs Google's 1,678 MB G5 int8 build (0.52×).
• G6: 4096 context, against the 1280 of Google's own G6 bundle (3.2× the context), at about the same size (~2.1 GB vs 2.04 GB).

---

⚠️ How to run it

These are precompiled DarwiNN DGCs. They need a GoogleTensor dispatch runtime that matches your chip's generation.

👉 G3/G4: use Box. It ships the correct runtimes/dispatch libs. Download the APK, put the .litertlm for your SoC on the phone, load it with the NPU backend.

Edge Gallery import settings

Use Import Model with: max tokens 4096, accelerator NPU only, image/audio/thinking/speculative decoding off.

---

Performance (Pixel 9 Pro XL, Tensor G4)

Measured 2026-09-10 on the G4 bundle:

Prefill scales, decode is pinned (~16 tok/s, dispatch-bound): fastest option on the phone for prefill-heavy work (RAG, long context, summarization, classification).

---

The exact AOT recipe (reproducible)

Full step-by-step for both paths in AOT_RECIPE.md. The int4 path has an executable version with guardrails in recipe/gemma3_int4hr_recipe.py.

Shared:

1. Source: QAT weights google/gemma-3-1b-it-qat-int4-unquantized (not plain -it).
2. Export: litert_torch export_hf: split_cache=True, cache_length=4096, prefill_lengths=[128], externalize_embedder=True. Use the float prefill_decode (model.tflite), not the default int8 copy that export_hf also writes.

G3/G4 (int4 + Hadamard):

1. Quantize: ai_edge_quantizer dynamic_wi4c_hr_afp32 (int4 per-channel + Hadamard rotation), applied once.
2. Rank-2 FC surgery: recipe/fc_rank2_surgery.py (see RANK2_FC_SURGERY.md).
3. Compile: large-model-support only, no sharding flag. ai_edge_litert 2.2.0 + Google Tensor beta compiler (BuildID 87ea807a…).

G5/G6 (int8):

1. Quantize: ai_edge_quantizer dynamic_wi8_afp32 on the float model.tflite.
2. No rank-2 surgery.
3. Compile: large-model-support only. Tensor SDK v2.1: ai-edge-litert 2.3.0 + ai-edge-litert-sdk-google-tensor 2.3.0 (compiler BuildID f4df3004…). In our tests the nightly packages INTERNAL'd on this graph and the stable 2.3.0 packages compiled it.

Reproducibility check: a from-scratch G4 rebuild with this recipe produces a compiled prefill_decode that is bit-identical to the published G4 build (md5 f1102df2…), and gives word-for-word identical replies on device.

Quantization notes

• G3/G4: Hadamard rotation is the coherence fix. Plain per-channel int4 is fluent-but-wrong; the same int4 + Hadamard is correct. Also what lets the 4096-context graph compile.
• G6: int4 is not yet coherent with the current compiler. All 8 int4 variants we tried word-salad on retail silicon. int8 is coherent.
• int4 per-tensor compiles but word-salads (one scale over the 262k-vocab head).
• Blockwise int4 compiles but is incoherent on the NPU AOT path.
• Per-quant support is compiler-build-dependent: retest against your own compiler instead of trusting a table.

---

Files

---

License & attribution

Base model Gemma 3 1B-IT © Google, under the Gemma Terms of Use. These quantized/repackaged derivatives inherit the Gemma license. Runtime: LiteRT-LM / LiteRT (Apache-2.0). Quantization, rank-2 FC surgery, int4 + Hadamard and int8 AOT compiles, and Tensor NPU packaging by @xThr45hx. Not affiliated with or endorsed by Google.