qwen-vknn

Qwen2.5-Coder-0.5B — VKNN .vxm (GPU, on-device)

Qwen2.5-Coder-0.5B (with-past decoder) compiled to VKNN's .vxm format for on-device GPU inference via the Vulkan backend.

Which file? For on-device chat, use qwen2.5-coder-0.5b-instruct-c1024-prefill256-int4.vxm — the smallest (~517 MB) with the fast prefill bucket; the VKNN app-demo ships it. The fp16 -prefill256 and single-bucket instruct-c1024 files trade size for full fp16 quality. The two decode-* files are raw base-completion models (no ChatML), for text continuation rather than chat. The tokenizer files serve both the Python front-end (tokenizer.json, via AutoTokenizer) and the app's byte-level BPE (vocab.json + merges.txt).

Instruct variants answer ChatML-formatted prompts; stop on <\|im_end\|> (id 151645).

The prefill bucket processes the prompt in one 256-token forward instead of token-by-token, cutting time-to-first-token by an order of magnitude on long prompts; the decode bucket then generates with the engine-resident KV cache. vknn_chat selects the buckets automatically (single-bucket files keep the token-by-token path). Two-bucket files need a VKNN build with multi-bucket prompt prefill (2026-07 main or later).

Two-bucket compile (fp16 shown; the int4 file adds -Os --quant-samples 0 instead of --fp16):

vknn_compile out.vxm --fp16 \
  --graph "model.onnx;dim:sequence_length=256;dim:past_sequence_length=1024;dim:total_sequence_length=1280" \
  --graph "model.onnx;dim:sequence_length=1;dim:past_sequence_length=1024;dim:total_sequence_length=1025"

Single-bucket compile:

vknn_compile model.onnx qwen2.5-coder-0.5b-instruct-c1024.vxm --fp16 -O1 \
  --dim sequence_length=1 --dim past_sequence_length=1024

Each fp16 model's greedy decode stream matches its HuggingFace fp32 reference token-for-token; the int4 variant stays coherent under the same ChatML contract (weight-only int4 with per-layer error guards). The VKNN app-demo on-device chat app ships the instruct model wrapped in ChatML.