Qwen3.5-9B-npu

Qwen3.5-9B — every component on the Intel NPU

Now also on the shared engine onw (俺のNPUがこんなに動くわけない): one engine for LFM2, Qwen3.6-35B-A3B, Qwen3.5-9B and Gemma 4 — start.bat ryugyosoft/Qwen3.5-9B-onw. This repo stays the standalone version.

Qwen/Qwen3.5-9B (text + image) running with all model components on the Intel NPU: vision encoder, text embeddings, the 32-layer hybrid decoder (24 Gated DeltaNet + 8 full-attention layers) and the LM head. The host only tokenizes, preprocesses pixels, computes MRoPE position ids, builds the attention mask, copies KV/state rows and samples. Comes with a browser chat UI and an OpenAI-compatible API.

Not llama.cpp and not OpenVINO GenAI's pipeline: the graphs were rewritten for the NPU (the DeltaNet Loops are replaced by matrix forms, the vision tower is fused into one static graph) and a small generation loop manages the caches. NPU execution itself is OpenVINO. Sister repo: ryugyosoft/gemma-4-E4B-it-npu.

Tested on two machines: Core Ultra 9 285HX (Arrow Lake HX, NPU 3720), Windows 11, NPU driver 32.0.100.5540, OpenVINO 2026.5.0b1 (nightly); and Core Ultra 5 226V (Lunar Lake, NPU 4000), Ubuntu 26.04, Linux NPU driver 1.38.0, OpenVINO 2026.5.0 nightly. Other NPUs are untested.

Lunar Lake (NPU 4000), measured with v1 (v2 not yet re-measured there): decode 7.8 tok/s, vision 0.37 s, prefill 7.3 s for 286 tokens; first start 187 s, later ~15-21 s. compare_devices.py: same 12 greedy tokens as the CPU, hidden-state error 3-9% (1-3% on NPU 3720). Loading takes ~9.5 GB of the host RAM (the NPU shares it), so a 16 GB machine has little room left.

$ python qwen35_npu.py cat.jpg "この画像に何が写っていますか？日本語で簡潔に説明してください。"
[load] 11.5s on NPU
この画像には、箱の中でリラックスして寝ている猫が写っています。
[perf] vision 862 ms | prefill 286 tok 13829 ms | decode 15 tok 4.3 tok/s

New in v2

The previous version is kept as the v1 tag: hf download ryugyosoft/Qwen3.5-9B-npu --revision v1 --local-dir ....

• Chunkwise-parallel DeltaNet for prompt blocks. Instead of 16 (or 64) sequential recurrence steps, a block is computed with a few small matrix products (below). 16-token block: 562 → 348 ms; with the 64-token block prompts prefill 3.6x faster than v1.
• 64-token prefill blocks (10.5 ms/token vs 23 with 16-token blocks). The extra decoder variant holds another ~3.5 GB, so it is used only with ≥ 24 GB RAM; Q35_S64=1 / Q35_S64=0 forces it on / off.
• Prompt lookup decoding for greedy requests (temperature 0, the UI default): tokens that followed the same n-gram earlier in the conversation are checked as a draft in one 16-token pass. DeltaNet state cannot be rewound, so it is snapshotted before the check and, after a partial match, the accepted tokens are re-fed at the front of the next block. Code edit 4.7 → 6.6 tok/s, summary 4.6 → 8.9 tok/s; free chat ~10% slower in the worst case (misses back off exponentially). No draft model, no extra memory.
• Prefix reuse: the recurrent/conv/KV state at the end of the previous conversation is kept, so a follow-up turn only processes the new messages (usage.prompt_tokens_details.cached_tokens).
• The 1-token decoder uses a matrix form of the recurrence (191 → 182 ms/token).

All outputs were checked against the plain greedy decoder / original model (compare_prefill.py, test_pld.py, test_multiturn.py).

Quick start (one click)

1. Download: hf download ryugyosoft/Qwen3.5-9B-npu --local-dir qwen35-npu (Python 3.10+ required).
2. Start: Windows double-click start.bat, Ubuntu bash start.sh.
3. A browser opens the chat UI at http://localhost:8000/ (attach / paste / drop an image; "思考モード" toggles thinking).

The first start installs packages into .venv (~2 GB) and compiles for the NPU (up to ~40 min on NPU 3720); later starts ~20 s. Without an NPU (or its driver) everything falls back to the CPU. The OpenAI API is at http://localhost:8000/v1.

python qwen35_npu.py photo.jpg "この画像を説明して"          # CLI (add --think for thinking mode)
python qwen35_npu.py - "Explain photosynthesis briefly."       # text only
python server.py --port 8000 --open [--device CPU]             # chat UI + OpenAI-compatible API
python example_client.py photo.jpg                             # official OpenAI SDK against the server

API: GET /v1/models, POST /v1/chat/completions with stream, stream_options.include_usage, max_tokens, temperature, top_p, seed, stop, multi-turn messages, image_url parts (data: or http(s)), and chat_template_kwargs: {"enable_thinking": true} for thinking mode. Requests are served one at a time.

For the NPU on Ubuntu install Intel's Linux NPU driver (https://github.com/intel/linux-npu-driver/releases, Ubuntu 24.04/26.04), add yourself to the render group and check ls /dev/accel/accel0.

Limitations

• One image per conversation, resized to a fixed 512x512 (aspect ratio is not kept). No video.
• Static 1024-token context (an image is 256 tokens). Thinking mode spends a lot of that budget.
• Prefix reuse keeps only the most recent conversation (a different chat starts from scratch).
• Needs a pre-release OpenVINO (2026.5 nightly) and transformers 5.2.
• On NPU 4000 the decoder is compiled per layer through NPUW (as one graph it does not finish compiling); qwen35_npu.py and compare_devices.py pick this from the NPU architecture, Q35_LM_CONFIG (JSON) overrides it.
• NPU workarounds were found on NPU 3720; compare_devices.py (NPU vs CPU) and qwen_layer_probe.py (per layer) show whether another NPU computes the same numbers.
• The multi-token-prediction (MTP) head of Qwen3.5 is not used: speculative decoding with it was exact (73% of drafts accepted) but only +9% end to end on NPU 3720 for ~5 GB more RAM, too much for 16 GB machines.

What is in this repo

Weights: INT4 symmetric (group 128) with INT8-symmetric backup, exported with optimum-intel from the original checkpoint (NPU 3720 cannot dequantize asymmetric INT4).

How it works

Gated DeltaNet on the NPU. OpenVINO exports each of the 24 linear-attention layers as a Loop that applies one step of the gated delta rule per token:

S  <- S * exp(g_t)                       S: [heads, 128, 128] recurrent state
kv <- S^T k_t
S  <- S + k_t (x) ((v_t - kv) * beta_t)
o_t <- S^T q_t

NPUs cannot run Loop. The 1-token decoder writes the step as matrix products: [q;k]^T S in one MatMul, o = d·q^T S + (q·k)·delta, S' = d·S + k delta^T (qwen_fixes.unroll_deltanet_loops(form="matmul")). Prompt blocks of T tokens use the chunkwise-parallel form (qwen_fixes.chunk_deltanet_loops): with G = cumsum(g) and the incoming state S0,

N   = diag(beta) [exp(G_t - G_j) k_t.k_j]_{j<t}          strictly lower triangular
D   = (I + N)^-1 diag(beta) (V - diag(exp G) K S0)         per-token deltas
O   = diag(exp G) Q S0 + [exp(G_t - G_j) q_t.k_j]_{j<=t} D
S_T = exp(G_T) S0 + K^T diag(exp(G_T - G)) D

The state is read and written once per block instead of twice per token. The triangular inverse is computed by exact block doubling, (D + C)^-1 = D^-1 - D^-1 C D^-1, where every intermediate is itself a true sub-inverse. The textbook series (I - N)(I + N^2)(I + N^4)... is also exact, but with real (highly correlated) keys its terms grow like binomial coefficients and cancel catastrophically in the NPU's fp16 (400% error vs 0.1%). A token_valid input (g = 0, beta = 0 for padding) lets a partly filled block leave the state exactly as after its last real token, so prompt tails need no 1-token steps. All decoders share the caches, which live in the NPU request's own input buffers: 16 KV tensors (appended), 24 conv states and 24 recurrent states.

fp16 precision of the DeltaNet output. On the NPU the decoder first disagreed with the CPU by 30-100% from the very first token. A per-layer probe traced it to the DeltaNet output: per-head RMS is only 1e-3..2e-2, and the small heads lost their precision in fp16; the gated RMSNorm right after blew that up to ~25% error per layer. The output only feeds that scale-invariant norm, so q is multiplied by 1024 inside the recurrence and the norm's eps by 1024^2 (qwen_fixes.scale_deltanet_output): same math, values out of the subnormal range. Every layer is now within 2% of the CPU, and generations match token for token.

Vision as one static graph. The export splits the vision tower into patch embedding, a learned position table interpolated bilinearly (4 corner indices + weights computed on the host), and the ViT + merger fed with host-built rotary embeddings and a block attention mask. For a fixed 32x32-patch grid all of those are constants, so make_static_vision_qwen.py computes them once and fuses the three graphs into pixel_patches[1024,1536] -> image_tokens[256,4096]. MRoPE position ids for the language model come from transformers' get_rope_index.

Rebuilding

optimum-cli export openvino -m Qwen/Qwen3.5-9B --task image-text-to-text --weight-format int4 --sym --group-size 128 --ratio 1.0 --backup-precision int8_sym models/Qwen3.5-9B-int4-sym-ov
python make_static_qwen.py models/Qwen3.5-9B-int4-sym-ov models/lm_static 1 1024     # matmul form
python make_static_qwen.py models/Qwen3.5-9B-int4-sym-ov models/lm_static 16 1024    # chunk form (default for S > 1)
python make_static_qwen.py models/Qwen3.5-9B-int4-sym-ov models/lm_static 64 1024
python make_static_vision_qwen.py models/Qwen3.5-9B-int4-sym-ov models/vision_static 32 32

(export with transformers 5.2 and optimum-intel from git.)

日本語での概要

Qwen3.5-9B（画像＋テキスト）の 全部品を Intel NPU 上で動かす 一式です。24層の線形アテンション（Gated DeltaNet）は OpenVINOの変換では Loop 演算になっていてNPUで動かないため、行列演算の形（プロンプトはチャンク並列形式）に書き換えました。 さらにDeltaNetの出力がFP16で精度を失う問題（1層あたり約25%のずれ）を、数学的に等価な倍率変更で解消し、 CPUと同じ生成結果になっています。Windowsは start.bat、Ubuntuは bash start.sh でブラウザのチャット画面が開きます。 NPU 3720で生成 4.6〜4.8 tok/s。画像は1会話1枚・512×512固定です。 v2 では、DeltaNetのチャンク並列化と64トークン単位のプリフィル（RAM 24GB以上）で画像付きプロンプトの処理が約14秒→3.9秒に、 会話中の文字列を再利用する回答はプロンプトルックアップデコーディングで最大約2倍に、2ターン目以降は差分だけを処理する プレフィックス再利用で応答開始が速くなりました。いずれも出力は通常のグリーディ生成と同じです。

License

Apache 2.0, same as the base model Qwen/Qwen3.5-9B. The weights here are re-quantized and restructured from it.