Qwen3.5-2B

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 17.5 tok/s · prefill 450 tok/s · TTFT 520 ms · all 25703 ops delegated (2026-09-05); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 19.0 tok/s · prefill 181 tok/s · TTFT 1.20 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 4.2 tok/s · prefill 56 tok/s · TTFT 5.04 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen35-2b-vl-int8/CARD.md

Qwen3.5-2B — LiteRT-LM

Qwen/Qwen3.5-2B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0; the 2026-10-16 text file checked on litert-lm 0.18.0 and 0.15.0 on a Mac). Same conversion rail as our Qwen3.5-0.8B and Qwen3.5-4B — the GPU-delegable rank-≤4 gated-delta kernel.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 2B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal, and this repo now carries both halves: the original text-only build (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does), and a vision build that adds the checkpoint's own 24-layer ViT. See Vision build below.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

2026-10-11: Qwen3.5-2B_int8.litertlm re-quantized — the 36 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) stay float (+3.4 MB), everything else byte-identical to the previous quantization; this removes the CPU repetition loop on multi-step arithmetic at the default (thinking off). Measured in the Honest notes below. Previous file: sha256 8ae50271…, 2,116,592,816 B; the 2026-10-11 file: sha256 e7473f57….

2026-10-11 (VL): Qwen3.5-2B-VL_int8.litertlm re-quantized the same way — its decoder section carries the same 36 float projections (+3.5 MB); the embedder, vision encoder, vision adapter, tokenizer and metadata sections are byte-identical to the previous file. Previous file: sha256 f8821e40…, 3,148,905,040 B; this file: sha256 33d9d901…, 3,152,378,448 B. Measured in the Vision build's Honest notes below.

I am sorry: anyone who pulled Qwen3.5-2B_int8.litertlm (the 2026-08-14 file) or Qwen3.5-2B-VL_int8.litertlm (the 2026-08-25 file) before the 2026-10-11 re-quantization and ran the CPU backend could get looping replies to multi-step arithmetic at the default (thinking off). On the GPU backend, the same files answered the arithmetic prompt correctly. The one-Engine line below was added to this card only on 2026-10-12, although my open report on the runtime had described that behaviour since 2026-08-06. This was my recipe's defect and my omission. From now on, when one file is found defective, the other files made with the same recipe are re-checked (the 0.8B files were re-checked and re-quantized on 2026-10-12), and this card's mechanical check before publication also covers the one-Engine line and a measured arithmetic result on the CPU backend.

2026-10-16: the tokenizer section of Qwen3.5-2B_int8.litertlm replaced with the repository's own tokenizer.json (Qwen/Qwen3.5-2B at 15852e8c, 12,807,982 B, sha256 5f9e4d49…); the TFLite section, the executor metadata and the LLM metadata (chat template included) are byte-identical. The previous section was the tokenizer as transformers writes it for the Qwen2Tokenizer class (19,989,325 B, sha256 06b95093…), whose pre-tokenizer has no \p{M} class, so text with combining marks was split differently from the repository's tokenizer: 49 tokens instead of 36 on a Hindi test sentence, 24 instead of 12 on Thai, 29 instead of 16 on Tamil, 30 instead of 29 on Arabic with harakat. English, numbers, code, chat markup, Chinese, Japanese, Korean and emoji tokenize the same, and so does every prompt of the checks and benchmarks on this card. The previous section also held the seven audio / TTS markers of tokenizer_config.json (<|audio_start|> … <|audio_pad|>, ids 248070–248076) as single tokens; the repository's tokenizer.json does not list them, so this file reads them as plain text. Previous file sha256 e7473f57…, 2,120,039,328 B → this file sha256 fa4ac325…, 2,119,744,416 B. Qwen3.5-2B-VL_int8.litertlm is unchanged: every version of it has carried the repository's tokenizer.json. The checks run on the new file are the 2026-10-16 item in the Correctness list; every other result on this card was measured on earlier files, as each item says, and where an item dated before 2026-10-16 says "this file" or "the previous file", it means the file of its date or the one before that; the text file of 2026-10-11 has the TFLite section of the current file byte for byte. I am sorry: every version of the text file since 2026-08-14 carried that tokenizer, so text in Hindi, Thai, Tamil, Arabic with harakat or another script with combining marks reached the model as other token ids than the repository's tokenizer gives (what that did to replies was not measured), and my checks, whose prompts tokenize the same with both, did not show it.

Correctness

All gates below were run on earlier files of this name (or their float parent, same graph rail), except the 2026-10-16 item, run on the current file:

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): on the composite 8-question probe (all 8 questions in one 138-token prompt), the GPU output matches the HF fp32 reference word-for-word through answer 7 (including reproducing the model's own arithmetic slip on question 1 — faithful conversion includes the model's mistakes). At that point fp32 ends its turn without attempting question 8; the int8 path diverges by that one end-of-turn token and adds the correct 8th answer before stopping.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.
• One Engine per conversation. A new Conversation opened on an Engine that already ran another conversation continues from that conversation's state: it opened its replies the way its own system prompt asked in 0 of 9 turns, against 9 of 9 on a fresh Engine (3 runs, litert-lm 0.18.0, Mac CPU). Create a new Engine for each conversation — the runtime reuses the prompt tokens a new conversation shares with the previous one and does not reset the recurrent state (my open report); a process that runs one conversation is unaffected.
• Tokenizer section (2026-10-16, the current text file, Mac M4 Max). The runtime's Engine.tokenize gives the same ids as the tokenizers library with the repository's tokenizer.json on 174 of 174 test strings (21 sentences, Hindi, Thai, Tamil and Arabic with harakat included, and every prompt of the checks on this card) on litert-lm 0.18.0 (GPU) and 0.15.0 (CPU). The 8-question sanity gate answers 8/8 on CPU and on GPU (litert-lm 0.18.0, one fresh process per question), a three-turn conversation completes on the CPU, every GPU subgraph is delegated with no validation error, and litert-lm 0.15.0 answers a plain prompt on the CPU.

Usage

litert-lm run ./Qwen3.5-2B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-2B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: the full 1–1024 ladder (1024, 512, 256, 128, 64, 32, 16, 8, 4, 2, 1). Every exported signature is charged engine memory even if never called; unlike the 4B (which needs a reduced ladder to stay under a 12 GB iPhone's jetsam ceiling), the 2B's full ladder peaks at ~5.3 GB on iPhone 17 Pro GPU.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

The Mac and iPhone rows were measured on the 2026-08-14 file (sha256 8ae50271…), the same graph with the 36 GatedDeltaNet decay/gate projections int8; the Galaxy S26 rows further down are of the 2026-10-11 file (sha256 e7473f57…). Every row of the text build was measured before the 2026-10-16 tokenizer replacement; the Galaxy S26 rows on the 2026-10-11 file, whose TFLite section the current file carries byte for byte.

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

(iPhone GPU: model load ~60 s on first launch while Metal programs compile.)

Galaxy S26 (Android, Qualcomm SM8850), the 2026-10-11 file (sha256 e7473f57…, the TFLite section of the current file), prebuilt litert_lm_advanced_main 2026-09-18, --benchmark_prefill_tokens=256 --benchmark_decode_tokens=64 --num_iterations=3, median of iterations 2–3, caches removed before the warm-up run (engine init = that cold run), phone locked and rested until its clocks were uncapped:

The S26 pulls its CPU clocks down within about 8 seconds of a sustained CPU run (the first 3 of 18 two-second samples uncapped, then 4.65 → 2.44 GHz by the end of the 37 s bench), so the CPU row is what a sustained run gets on this phone; the GPU stayed at its 1300 MHz ceiling throughout. The file before it (sha256 8ae50271…) under the same protocol on the same phone: CPU 562.1 / 22.8 tok/s, within 1 % (the 36 float projections cost nothing measurable), and its CPU reply to the arithmetic probe repeated $391$ to the token cap. Every GPU process on this binary pays the full engine init: a cache written by one process fails to deserialize in the next (Deserialization failed: DATA_LOSS) and the engine recompiles.

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and, on Android, GPU verified on the Galaxy S26 (Adreno) only; no Mali phone — the Pixel 8a (Arm Mali, OpenCL) is not verified on this file, see its bullet below. On the Galaxy S26 (OpenCL, 2026-10-11, the rows above) every signature was fully delegated and three fresh processes answered the arithmetic probe correctly. Those rows put the CPU ahead in every column.
• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (8 GB-class Android): GPU does not fit. Until 2026-10-12 this card listed the Pixel 8a as a verified GPU device — my error, and I am sorry to anyone who chose the GPU on a Mali phone because of it: on the Pixel 8a the engine build drains the phone's memory. On 2026-10-12, with the same prebuilt binary as the S26 rows above, GPU engine creation for the 2026-10-11 file (the TFLite section of the current file) reached 2 of the 11 prefill signatures, fully delegated each one, and took the phone's free memory from 4,137,276 KiB towards our test's stop line of 400,000 KiB. Our test reads free memory every 2 s and stops the run at the first reading under that line. In this run that reading was 253,700 KiB, after Android had begun closing other apps. Whether the build completes without the stop is not measured. On Android, use the CPU backend: on the 11.4 GB Galaxy S26 the GPU engine builds, but the CPU row above is better in every column. Before it is published, this card is now checked mechanically for the device limits measured on its files.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality (litert-lm 0.18.0, one fresh process per run): the 8 short gate questions answer 8/8 on CPU and GPU. On multi-step arithmetic at the default (thinking off) the previous int8 file looped or repeated on CPU — What is 17 times 23? gave 340 + 17 = 340 + 17 = … in 3 of 3 processes on a Mac M4 Max and 391 followed by $391$ repeated to the cap on a Pixel 8a — while the same file answered 391 on GPU and the fp32 export answered 391 on CPU. The 2026-10-11 file answers the six digit questions 17 of 18 times on Mac CPU (the 18th is a correct derivation cut by a 400-token cap; uncapped, 3 of 3), 3 of 3 cleanly on the Pixel 8a CPU, and on the Galaxy S26 in every fresh process run (GPU 3 of 3, CPU 4 of 4, the same reply text byte for byte); see the note below. The composite 8-question probe of the earlier card text was not re-run on this file.
• 2026-10-11 re-quantization. Cause of the CPU loop: LiteRT-LM's CPU path quantizes every int8 FC's input row to int8 at run time (XNNPACK dynamic quantization), and the GatedDeltaNet decay/gate heads (in_proj_a, in_proj_b, 16 outputs each) could not take that noise; keeping only those 36 projections in float (+3.4 MB) restores the fp32 answer, while float down_proj, float attention projections or per-channel smoothing did not. Everything else is the previous quantization byte for byte (same float export, same recipe, same template). Previous file: sha256 8ae50271…, 2,116,592,816 B.

Vision build (Qwen3.5-2B-VL_int8.litertlm)

The upstream checkpoint ships a 24-layer, 1024-dim ViT that the text-only package drops. This build keeps it, wired to LiteRT-LM's fast_vlm contract: one 512×512 image per turn, 1024 patches merged 2×2 into 256 soft tokens injected at the image position. There is no DeepStack in this tower (deepstack_visual_indexes is empty upstream), so the single image embedding is the whole visual signal — nothing is folded away.

litert-lm run ./Qwen3.5-2B-VL_int8.litertlm \
  --prompt "What is in this image?" --attachment ./photo.png

# vision on the GPU as well as the decoder
litert-lm run ./Qwen3.5-2B-VL_int8.litertlm --backend gpu --vision-backend gpu \
  --prompt "Where is this scene?" --attachment ./photo.png --cache no

Text-only prompts work on this file too; it is a superset of the text build, at a larger download and a larger GPU footprint.

Measured

The Mac and iPhone rows below were measured on the 2026-08-25 file (sha256 f8821e40…), the same graph with the 36 decay/gate projections int8; the Galaxy S26 rows in the Honest notes are of this file (2026-10-11).

Mac, litert-lm benchmark (0.16.0), M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine — text path:

iPhone 17 Pro, cold start, single runs, quality-gate harness — the vision rows are real image turns:

Grounding on device was checked with two probes whose answers are opposite: an image containing text ("does this image contain text?" → "Yes, the image…") and one without ("No, the image…"). Both correct on Metal, with the vision encoder on the GPU.

Honest notes

• Six prefill signatures, not eleven. Every exported signature is charged engine memory whether or not it is called. The text build's full 1–1024 ladder peaks at ~5.3 GB on iPhone GPU and fits; adding the vision tower pushes the same ladder past a 12 GB phone's jetsam ceiling — all three Metal legs were killed after initialising 11 of 12 signatures. This build ships 1024, 256, 64, 16, 4, 1. Prompts longer than a signature are chunked as usual, so the only cost is a slightly coarser chunk plan. Cutting the ladder also tripled CPU decode (3.8 → 14.1 tok/s) and took first-token latency from 23.7 s to 5.1 s, because the runtime's XNNPACK repack is charged per signature too.
• Positions are 1-D. The fast_vlm contract feeds sequential positions, so the checkpoint's M-RoPE collapses to plain RoPE. Against the full M-RoPE reference on nine image/prompt pairs, one generation is identical and the other eight are fluent same-content paraphrases that diverge deep in the answer. Expect that class of difference — counting and dense-layout questions are where it shows.
• The vision graph is faithful; the int8 damage is in the encoder. The exported tower matches the model's own vision path at correlation 0.9999999992 in fp32. Quantizing the encoder to int8 drops that to 0.97–0.98 on real photographs, so this build ships the encoder in fp16 and only the adapter in int8. An int8-encoder variant is 300 MB smaller and still answers correctly, but it is not what ships here.
• Android: measured on a Galaxy S26 (Qualcomm SM8850, Adreno) with this file, text turns (2026-10-11). Prebuilt litert_lm_advanced_main 2026-09-18, --benchmark_prefill_tokens=256 --benchmark_decode_tokens=64 --num_iterations=3, median of iterations 2–3, caches removed before the cold run: GPU (OpenCL, full delegation on every decoder signature) prefill 598.0 tok/s, decode 17.3 tok/s, TTFT 0.49 s, cold engine init 53.7 s (the prebuilt kit recompiles the GPU program in every process), peak 3.31 GB; CPU (XNNPACK) prefill 567.1 tok/s, decode 23.0 tok/s, TTFT 0.5 s, cold engine init 8.4 s (5.8 s with the runtime's cache), peak 2.71 GB — the S26 pulls its CPU clocks down within seconds of a sustained CPU run (6 of 8 sampler lines capped), so the CPU row is what a sustained run gets on this phone; the previous file measured 534.1 / 24.2 tok/s on the same CPU leg right after it. Image turns were not measured on the phone. On the Pixel 8a (Arm Mali, OpenCL) on 2026-10-12, GPU engine creation for this file (text turns) reached 2 of the 6 prefill signatures, fully delegated each one, and took the phone's free memory from 4,295,576 KiB towards our test's stop line of 400,000 KiB. Our test reads free memory every 2 s and stops the run at the first reading under that line. In this run that reading was 223,656 KiB. Whether the build completes without the stop is not measured. On Mali an fp16 vision encoder is known to crash devices of this class — an int8-vision build would be the Android path there.
• 2026-10-11 re-quantization (CPU). The previous VL file (sha256 f8821e40…) had the text build's CPU repetition loop: What is 17 times 23? at the default (thinking off) ran 340 + 17 = 340 + 17 = … to the token cap in 3 of 3 fresh Mac M4 Max CPU processes and in the Galaxy S26 CPU process, while the GPU answered 391. Same cause and same fix as the text file (its 2026-10-11 note above): the 36 GatedDeltaNet decay/gate projections stay float in the decoder section, everything else byte-identical. Measured on this file (litert-lm 0.18.0 on the Mac, one fresh process per run): the six digit questions 18/18 on Mac CPU (three runs each) and 6/6 on Mac GPU, the 8 gate questions 8/8 on CPU and 8/8 on GPU, thinking on (raw <think> prefill, no budget) 3/3 close the thought and answer 391 on Mac CPU, the image probe grounded on CPU and on GPU (the same sentence on both backends); Galaxy S26 CPU 4/4 and GPU 3/3 fresh processes answer 391 without repetition.
• Quality on the composite 8-question probe, separated by cause. On Mac this file answers 7 of 8, missing only 17 + 25 — and it misses it identically on GPU and CPU, so that one is the model, not the conversion (the text build's card records the same slip). On iPhone Metal two further answers degrade that do not degrade on Mac. Individual questions asked one at a time gate 8/8 on both Mac backends. Measured on the 2026-08-25 file; the composite probe was not re-run on the 2026-10-11 file.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B/4B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• Vision tower (VL build only): the upstream tower is dynamic-resolution, and its grid_thw preprocessing aborts torch.export outright. It is re-authored for one static 512×512 image — Conv3d patch-embed folded to Conv2d over the duplicated temporal frame, learned position embeddings bilinearly resampled to the static grid ahead of time, 2-D rope precomputed, explicit full attention instead of the variable-length split. Patches stay in raster order through the encoder and the 2×2 merge happens in the adapter as four strided slices and a concat, so no GATHER_ND reaches the mobile GPU delegate. Two numerical guards matter: every activation keeps a leading batch dim (Metal computes rank-2 elementwise against a rank-2 constant wrong, silently), and LayerNorm is pre-scaled by a calibrated power of two because from block 11 on the residual stream carries values large enough to overflow fp16 accumulation.
• Pad guard on the externalised-embedder graph: fast_vlm moves the embedder into its own tflite, which removes token ids from the decoder graph — so the gated-delta prefill-pad guard, which keyed off input_ids != 0, silently stopped arming. It now derives the valid mask from position monotonicity instead, the same fallback upstream's own exportable module uses. The 40/40 prompt-length sweep is what proves it armed.
• Unlike the 4B, the 2B's linear-attention heads are ungrouped (16 key / 16 value), so the 4B's head-interleave rewrite never traces, and the full prefill signature ladder fits on-phone.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded. The rows were measured on earlier files of these names, before the 2026-10-11 re-quantization (and, for the text file, the 2026-10-16 tokenizer replacement).