Qwen2.5-Coder-1.5B-Instruct

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 10.1 tok/s · prefill 206 tok/s · TTFT 1.18 s · all 1243 ops delegated (2026-08-24); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 12.0 tok/s · prefill 84 tok/s · TTFT 540 ms · all 1243 ops delegated (2026-08-19); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 29.7 tok/s · prefill 106 tok/s · TTFT 2.14 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen25-coder-1.5b-int4/CARD.md

Qwen2.5-Coder-1.5B-Instruct — LiteRT-LM

Qwen/Qwen2.5-Coder-1.5B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

A 1.54B code model at 1.12 GB — small enough that a phone-class device runs it at conversational speed rather than merely running it. Code assistance is the use case that most needs that: it returns short outputs constantly, so decode rate is the experience.

A 3B sibling exists in this org — litert-community/Qwen2.5-Coder-3B-Instruct, 3.43 GB, converted by another contributor. Pick by what you are optimising: that one for capability, this one for speed and footprint. The two have not been benchmarked against each other on the same machine, so this card makes no claim about their relative quality.

Correctness

Two gates, because a general-knowledge check certifies nothing for a code model:

• 8-question sanity gate: 8/8 on the GPU backend (Mac, litert-lm 0.16.0).
• Code gate: 6/6. Six small functions — fib, reverse_words, is_prime, largest contiguous sublist sum, count_vowels, flatten — where the generated code is executed against assertions, not read. A task counts only if the code imports and every assertion holds.

Usage

litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm \
  --prompt "Write a Python function that returns the nth Fibonacci number. Code only."

# GPU
litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Qwen's ChatML markers including the vendor's default system prompt (You are Qwen, created by Alibaba Cloud. You are a helpful assistant.), which the upstream template inserts whenever no system message is supplied — rendering byte-identical to upstream for a single-turn request. A 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1).

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --cache no, quiet machine, serialized:

For scale, a 3B-class int4 bundle converted on the same day and measured on the same machine and protocol runs 1241 tok/s prefill / 86.3 decode / 0.233 s TTFT. Decode on these runtimes is memory-bandwidth-bound, so halving the weights is the most direct speed lever there is.

Honest notes:

• A Mac's GPU-vs-CPU ratio does not transfer to a phone. Here the GPU decodes 2.9× the CPU; on a Pixel 8a a 3B model's GPU and CPU decode land within 1% of each other, because a phone's CPU and GPU share the same LPDDR. On phone hardware the GPU's win is prefill and time-to-first-token.
• The embedding table is stored in its own bundle section. That is what keeps this file at 1.12 GB — with the table quantized inline it duplicates per prefill signature and the same weights come out at 2.53 GB.

Galaxy S26 — GPU and CPU

The published bundle runs on the Android GPU backend and generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_1024; the runtime accepts that split.

Speed on the same handset with one binary and one 205-token prompt file; only --backend changes (litert_lm_advanced_main from litert-lm 0.16.0, --benchmark, --sampler_backend=cpu). The GPU row is the gate run of 2026-08-24 (one run); the CPU rows were taken 2026-09-05, two cold runs (compile and weight caches deleted between runs) with the phone cooled below 42 °C before each run.

The CPU decodes 3.3× faster; the GPU wins prefill and peaks lower. A code answer is mostly decode (the bench replies ran 360–409 tokens), so this repo's litertlm_manifest.json recommends cpu for flagship-class Android on that pair. Pick gpu for long prompts with short answers.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0 — a pristine released stack, no patched checkout.

• EXTERNALIZE_EMBEDDER=1 is required, not cosmetic. Qwen2.5-Coder ties its embedding and lm_head, so a recipe asking for int4 linears and an int8 embedder describes one tensor two ways; the quantizer resolves it by copying the 151936×1536 table once per signature. Seven copies, 1.63 GB, 65% of the file — and no error anywhere. Externalising the table removes the conflict.
• The default system prompt is baked into the template. Upstream inserts it when the caller sends no system message; a plain ChatML template would drop it and put the model in a state it was not tuned in on every default request.
• Quantization: int4 blockwise-32 with OCTAV clipping on the linears, int8 on the embedding.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle. This repository is a community conversion and is not affiliated with Alibaba Cloud or the Qwen team.