Downloads · 30 days
4.1K
48% of all-time downloads
litert-community/Qwen3.5-4B
Qwen3.5-4B is a text generation model from litert-community. Use it when you need the model to write or continue text. It is set up for litert-lm. The card lists the license as apache-2.0.
LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmo…
Downloads · 30 days
4.1K
48% of all-time downloads
All-time downloads
8.6K
Public
Repo size
33.1 GB
Likes
6
Public
Click a slice to open those files.
.litertlm7.2 GB · 100%
From the Hugging Face model README
LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0; the 2026-10-13 int8 file checked on litert-lm 0.18.0 and the Android CLI build of 2026-09-18; the 2026-10-16 files checked on litert-lm 0.18.0 and 0.15.0 on a Mac). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.
Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).
| File | Recipe | Size |
|---|---|---|
Qwen3.5-4B_int8.litertlm | int8 dynamic on linears + embedding (convs, the delta rule and the 48 GatedDeltaNet decay/gate projections in_proj_a / in_proj_b stay float; re-quantized 2026-10-13), fp32 activations declared | 4.42 GB (4,418,793,712 B) |
Qwen3.5-4B_mixed_int4.litertlm | Mixed INT4: int4 blockwise-32 on linears, int8 on embedding + lm_head (convs and the delta rule stay float), fp32 activations declared | 2.75 GB (2,754,070,624 B) |
2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.
2026-10-09: metadata updated (chat template with a thinking switch, reasoning channel, supports_thinking); TFLite, tokenizer and executor metadata byte-identical.
2026-10-13: the int8 file re-quantized — the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) now stay float, and the rest of the recipe is unchanged; the chat template, tokenizer and executor metadata are byte-identical. Qwen3.5-4B_int8.litertlm: previous file sha256 a360b83d…, 4,407,428,464 B → the 2026-10-13 file sha256 ce4c2d64…, 4,419,088,624 B (+11,660,160 B). Its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. This fixes two things on the CPU backend, as measured in the Correctness list and the Honest notes below: replies on multi-step arithmetic that did not stop before the token cap, and, with thinking on without a budget, a thought that did not close. I am sorry: since 2026-08-14, anyone who pulled the int8 file could get a reply that ran on to the token cap on multi-step arithmetic on the CPU backend (7 of 18 runs of the six multiplication questions below, 1 of them with no answer) and, with thinking on without a budget, a thought that did not close (0 of 3 runs closed it); on 2026-10-12 this card's own check recorded a run that reached the cap and still reported the file as answering 3 of 3; and the one-Engine-per-conversation note reached this card only on 2026-10-12, though my own report behind it was filed on 2026-08-06. This was my recipe's defect and my check's miss. The Mac GPU did not show it: both files gave the right product and ended by themselves in 6 of 6 runs of the six questions (one per question), and both pass the 8-question sanity gate 8/8. From now on, when one file of a recipe is found defective, its sibling files are re-checked the same day with the six questions in three fresh processes each, a run that reaches the token cap counts as a failure even when its product is right, and this card's lines are checked mechanically before it is published.
2026-10-16: the tokenizer section of both files replaced with the repository's own tokenizer.json (Qwen/Qwen3.5-4B at 851bf6e8, 12,807,982 B, sha256 5f9e4d49…); the TFLite section, the executor metadata and the LLM metadata (chat template included) are byte-identical. The previous section was the tokenizer as transformers writes it for the Qwen2Tokenizer class (19,989,325 B, sha256 06b95093…), whose pre-tokenizer has no \p{M} class, so text with combining marks was split differently from the repository's tokenizer: 49 tokens instead of 36 on a Hindi test sentence, 24 instead of 12 on Thai, 29 instead of 16 on Tamil, 30 instead of 29 on Arabic with harakat. English, numbers, code, chat markup, Chinese, Japanese, Korean and emoji tokenize the same, and so does every prompt of the checks and benchmarks on this card. The previous section also held the seven audio / TTS markers of tokenizer_config.json (<|audio_start|> … <|audio_pad|>, ids 248070–248076) as single tokens; the repository's tokenizer.json does not list them, so these files read them as plain text. Qwen3.5-4B_int8.litertlm: previous file sha256 ce4c2d64…, 4,419,088,624 B → this file sha256 58288fd3…, 4,418,793,712 B; Qwen3.5-4B_mixed_int4.litertlm: previous file sha256 37209c42…, 2,754,365,536 B → this file sha256 7243c33c…, 2,754,070,624 B. The checks run on these files are the 2026-10-16 item in the Correctness list; every other result on this card was measured on earlier files, as each item says, and where an item dated before 2026-10-16 says "this file" or "the previous file", it means the file of its date or the one before that; the int8 file of 2026-10-13 and the Mixed INT4 files since 2026-08-27 have the TFLite section of the current files byte for byte. I am sorry: every version of these files since 2026-08-14 (int8) and 2026-08-27 (Mixed INT4) carried that tokenizer, so text in Hindi, Thai, Tamil, Arabic with harakat or another script with combining marks reached the model as other token ids than the repository's tokenizer gives (what that did to replies was not measured), and my checks, whose prompts tokenize the same with both, did not show it.
The Mixed INT4 variant exists for RAM-constrained devices (requested in LiteRT-LM #1658 for 8 GB Android phones): 2.75 GB of weights instead of 4.42 GB, measured ~1.7 GB CPU peak on the 138-token probe. If you have the memory for int8, int8 answers harder prompts slightly better (measured below).
The int8 results below, down to the GSM8K bullet, come from the previous quantization of the int8 file or its float parent (the same graph), except the One Engine item, run on the Mixed INT4 file, the 2026-10-13 item, run on the 2026-10-13 int8 file (sha256 ce4c2d64…, whose TFLite section and metadata the current file carries byte for byte), and the 2026-10-16 item, run on the current files:
<think> prefill, 400-token cap): 3 of 3 runs close the thought and answer 391.thought channel and answer 391 in a 70-character reply.litert_lm_advanced_main; thinking off by default, 400-token cap): the arithmetic prompt answers 391 in 4 of 4 runs, the same 483-character reply each time, with no repetition.Engine.tokenize gives the same ids as the tokenizers library with the repository's tokenizer.json on 174 of 174 test strings (21 sentences, Hindi, Thai, Tamil and Arabic with harakat included, and every prompt of the checks on this card) on litert-lm 0.18.0 (GPU) and 0.15.0 (CPU). The 8-question sanity gate answers 8/8 on CPU and on GPU (litert-lm 0.18.0, one fresh process per question), a three-turn conversation completes on the CPU, every GPU subgraph is delegated with no validation error, and litert-lm 0.15.0 answers a plain prompt on the CPU.Mixed INT4 correctness (all gates re-run on the Mixed INT4 file before 2026-10-16, with the TFLite section of the current file byte for byte):
litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."
# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."
Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).
Chat template note: both files carry the LiteRT-LM models/qwen3_5/chat_template.jinja at 87ab9334 with two edits: its format_content macro gains the tool_response branch of the current main version (it reads chat content as a string or as parts, text and tool_response), and enable_thinking defaults to false instead of true. With nothing set, the generation prompt opens each new assistant turn with an empty <think>\n\n</think> block (past assistant turns render without it), and a single-turn prompt (one user message, with or without a system prompt) renders byte for byte as before. enable_thinking=true pre-fills <think>\n; the reasoning is returned on the thought channel (<think>\n / \n</think>), and the text carries the answer only. Tested with litert-lm 0.18.0 on a Mac (CPU): with thinking on, both files return the reasoning on the channel and give 391 for 17 × 23. A two-turn conversation ("My name is Kenji." / "What is my name?") gets the name back with thinking unset and on; with thinking on, the int8 file's second reply greets the user by name ("Hello Kenji! It's great to meet you. …") instead of stating it. On 0.17.0 both files answer a plain prompt and a two-turn conversation. With thinking on, GPU and phones were not run; the updated mixed_int4 file answered a plain prompt on a Galaxy S26 (CPU, litert_lm_main built from the v0.18.0 tag); for the Mixed INT4 file, the card's GPU and iPhone results come from an earlier file, whose TFLite section and default prompt are byte-identical; for the int8 file, the Mac, iPhone and Raspberry Pi Performance rows come from the previous quantization (same default prompt), and the 2026-10-13 file's own checks are the 2026-10-13 item in the Correctness list. The template renders tool definitions, but the files' llm_model_type is generic_model, so the runtime does not parse the model's tool calls: they come back as text, and supports_function_calling is false.
The Mac, iPhone and Raspberry Pi int8 rows below were measured on the previous quantization of the int8 file (TFLite section sha256 4666cd03…, as in the files sha256 74abbd14… and a360b83d…), the same graph with the 48 decay/gate projections int8; the Galaxy S26 row further down is of the 2026-10-13 int8 file (sha256 ce4c2d64…). Every row in this section and in the Raspberry Pi section was measured before the 2026-10-16 tokenizer replacement; the Galaxy S26 row and the Mixed INT4 rows on files whose TFLite section the current files carry byte for byte.
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:
| Backend | Prefill (256) | Decode | TTFT |
|---|---|---|---|
| GPU | 672 tok/s | 62.0 tok/s | 0.40 s |
| CPU | 243 tok/s | 19.9 tok/s | 1.11 s |
On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|---|---|---|---|---|
| iPhone 17 Pro | GPU (Metal) | 87.1 tok/s | 9.3 tok/s | 1.93 s | 4.98 GB |
| iPhone 17 Pro | CPU | 24.0 tok/s | 5.4 tok/s | 6.88 s | 1.70 GB |
(iPhone GPU: model load 66 s on first launch.)
Galaxy S26 (Qualcomm SM8850), the 2026-10-13 int8 file (sha256 ce4c2d64…, the TFLite section of the current file), the LiteRT-LM Android CLI build of 2026-09-18 (litert_lm_advanced_main), CPU benchmark of 256 prefill and 64 decode tokens, 3 iterations, median of iterations 2–3, caches removed before the warm-up run (engine init = that cold run); peak memory is the peak system RAM use reported by the CLI:
| Device | Backend | Prefill (256) | Decode (64) | TTFT | Engine init (cold) | Peak memory |
|---|---|---|---|---|---|---|
| Galaxy S26 | CPU | 178.9 tok/s | 8.93 tok/s | 1.55 s | 17.9 s | 5,517 MB |
The phone capped its CPU clocks in every benchmark iteration (our frequency readings: 14 of 18 capped), so the row is this phone's capped speed. The file before it (sha256 a360b83d…), measured right after on the same phone with the same benchmark: 157.2 / 8.64 tok/s, TTFT 1.78 s, cold engine init 20.4 s, peak 5,501 MB, with 16 of 19 readings capped; its first iteration, 199.9 / 11.09 tok/s, is within 1 % of the 2026-10-13 file's 201.5 / 11.11. Four fresh processes of the 2026-10-13 file answered the arithmetic prompt (17 × 23) with 391, the same 483-character reply each time, with no repetition. Neither the 2026-10-13 nor the current int8 file was started on the phone's GPU; a Galaxy S26 hard-rebooted during the previous file's 2026-08-24 GPU attempt (see the Honest notes).
On a Pixel 8a (Tensor G3, 7.75 GB) CPU with the same Android CLI build, one process of the 2026-10-13 int8 file answered the arithmetic prompt (17 × 23) with 391 in the same 483-character reply, with a process peak of 5,407,976 KiB and the phone's free memory at or above 3,542,768 KiB (5,120,440 KiB before the run); no speed was measured.
Mixed INT4 (litert-lm benchmark 0.16.0, Apple M4 Max, same flags; iPhone rows are single cold runs of the 138-token quality probe, unplugged):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|---|---|---|---|---|
| Apple M4 Max | GPU | 669 tok/s | 68.5 tok/s | 0.40 s | — |
| Apple M4 Max | CPU | 100 tok/s | 20.1 tok/s | 2.61 s | — |
| iPhone 17 Pro | GPU (Metal) | 88.7 tok/s | 11.4 tok/s | 1.89 s | 5.74 GB |
| iPhone 17 Pro | CPU | 46.7 tok/s | 8.9 tok/s | 3.12 s | 1.68 GB |
| Pixel 8a (8 GB) | CPU (XNNPACK) | 22.1 tok/s | 6.3 tok/s | 11.9 s | 4.43 GB |
(iPhone GPU: model load 65 s on first launch. Pixel 8a: litert_lm_main v0.16.0, 260-token prompt / 204 decoded tokens, warm weight cache; peak is process VmHWM; engine init 30–50 s; TTFT scales with prompt length — 4.2 s on a 24-token prompt.)
Honest notes:
a360b83d…) could repeat one line to the token cap after its answer on multi-step arithmetic on the CPU backend (litert-lm 0.18.0, Mac M4 Max CPU, one fresh process per run, 400-token cap). With thinking off, the six multiplication questions (three runs each) got the right product in 17 of 18 runs, reached the cap in 7 of 18 (6 of them after the right product, 1 with no product) and ended by themselves in 11 of 18. With ThinkingConfig unset, the arithmetic prompt (17 × 23) answered 391 in 6 of 6 runs (two sets of three) and reached the cap after the answer in 2 of them. Thinking on without a budget (a raw <think> prefill): 0 of 3 runs closed the thought, each repeating one line. With a 300-token thinking budget: 3 of 3 runs returned the reasoning on the thought channel and began the answer with 391, and none of the 3 ended by itself (2 repeated a numbered **Final Answer:** 391. line to the cap, 1 was cut by the cap mid-sentence). On the Mac GPU the six questions got the right product and ended by themselves in 6 of 6 runs (one per question), and the 8-question sanity gate passed 8/8 on CPU and on GPU. On the Galaxy S26 CPU (the LiteRT-LM Android CLI build of 2026-09-18), 3 of 3 fresh processes answered the arithmetic prompt with 391, each ending by itself. The same cause and the same fix as the 2026-10-11 note on the Qwen3.5-2B card: LiteRT-LM's CPU backend quantizes every int8 FC's input row to int8 at run time (XNNPACK dynamic quantization), and the GatedDeltaNet decay/gate heads (in_proj_a, in_proj_b) could not take that noise. The 2026-10-13 file changes only those 48 projections, in the 24 linear-attention layers, to float (+11,660,160 B); its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. Measured on the 2026-10-13 file: Mac CPU 18 of 18 right with no repeated line, thinking on without a budget 3 of 3 closing the thought, with a 300-token thinking budget 3 of 3 answering 391, the 8-question sanity gate 8/8 on CPU and GPU, Mac GPU 6 of 6 and Galaxy S26 CPU 4 of 4.litert_lm_advanced_main 2026-09-18, 2026-10-12) the Mixed INT4 GPU engine builds with a cold init of 73–86 s and a benchmark peak of 3.8–4.1 GB, and one prompt run answered 17 × 23 with 391. The GPU's maximum clock fell from 1300 MHz to 646–726 MHz during the compile, so the speeds measured right after it are not published. The int8 file is not verified on an Android GPU: a Galaxy S26 hard-rebooted during its 2026-08-24 GPU attempt (the cause is not established; recorded in litertlm_manifest.json), and I have not run the int8 file on an Android GPU since, the 2026-10-13 file included; use the CPU. Until 2026-10-12 this bullet said GPU execution on Adreno was not verified; for the Mixed INT4 file, this S26 run changed that.ce4c2d64…, the TFLite section of the current file) answered the arithmetic prompt (17 × 23) with 391, in the same 483-character reply as on the Galaxy S26. The process took 146.8 s, including the first-run weight-cache build. It peaked at 5,407,976 KiB, and the phone's free memory stayed at or above 3,542,768 KiB (5,120,440 KiB before the run). Our test stops a run when the phone's free memory falls under 400,000 KiB; it did not stop this one. This is one process, and no speed was measured. This run corrects this bullet's earlier "not attempted, and not expected to fit", which was an inference.37209c42…, the TFLite section of the current file) on the Pixel 8a (Arm Mali, OpenCL) reached 1 of the 6 prefill signatures, fully delegated it, and took the phone's free memory from 4,244,000 KiB towards our test's stop line of 400,000 KiB. Our test reads free memory every 2 s and stops the run at the first reading under that line. In this run that reading was 139,628 KiB, after Android had begun closing other apps, so do not try GPU on 8 GB devices. Whether the build completes without the stop is not measured. Before it is published, this card is now checked mechanically for the device limits measured on its files.Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):
BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.[B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.ai-edge-quantizer dynamic int8 on the FULLY_CONNECTED layers and the embedding, with prefer_activation_type fp32. The convs and the delta rule stay float, and since 2026-10-13 so do the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b of the 24 linear-attention layers).ai-edge-quantizer on the same float export — int4 BLOCKWISE-32 min-max on every FULLY_CONNECTED, overridden to int8 channelwise on the lm_head and the embedding lookup (one shared 248320×2560 vocab buffer, kept single). Block size and algorithm were chosen by measurement: b32/b128 × min-max/OCTAV all built from the same export and scored on GSM8K n=100 (93 / 90 / 92 / 90) — blockwise-32 min-max won. Convs and the delta rule stay float, prefer_activation_type fp32 as in the int8 file.Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|---|---|---|---|
Qwen3.5-4B_int8.litertlm (earlier file, sha256 a360b83d…) | 20.2 (20.1–20.3) | 1.8 (1.8–1.8) | 13.2 s | 6.2 GB |
Qwen3.5-4B_mixed_int4.litertlm | 17.1 (17.0–17.2) | 2.4 (2.3–2.4) | 15.4 s | 4.9 GB |