Qwen3.5-9B-rk3588

Qwen3.5-9B for RK3588 NPU

This repository provides a hardware-accelerated port of Qwen3.5-9B optimized for Rockchip RK3588 NPU.

<br>User:<image>Describe the image.<br><br> Answer:The image depicts a humorous and surreal scene set on the surface of the Moon:

• An astronaut in a white spacesuit is reclining casually, as if relaxing or taking a break.
• The astronaut’s helmet reflects the Earth above them — showing a view of our planet from lunar orbit.
• In their gloved hand, they’re holding a green glass beer bottle (possibly resembling a Heineken).
• Their feet are propped up on what appears to be a cooler or crate labeled “Heineken” (though it’s stylized and hard to read clearly).
• Behind them, the Earth looms large in the sky — blue oceans, white clouds, and green landmass visible.
• To the right, part of a lunar ladder or structure can be seen, suggesting this might be near a space base or landing site.
• The ground is rocky and dusty, typical of the lunar surface.

The overall tone is whimsical and satirical — imagining an astronaut enjoying a beer on the Moon, blending science fiction with everyday leisure culture. It’s likely meant to evoke humor rather than realism.

Note: This image is not scientifically accurate — no such beverage or relaxation would be possible in actual space due to lack of atmosphere, extreme temperatures, radiation, and microgravity effects. But as art, it’s fun! 🌕🍺

---

Model Files

Hardware Requirements

• Rockchip RK3588 / RK3588S
• RKNPU2 driver
• Tested on:
  • Rock 5C
  • Ubuntu 22.04 / 24.04 (Joshua Riek)

Runtime Requirements

• RKLLM runtime (version 1.3.0)
• RKNN runtime (rknpu2)
• OpenCV (for image preprocessing)

Model performance benchmark (FPS)

All models, with C++ examples, can be found on the Q-engineering GitHub.<br><br> All LLM models are quantized to w8a8, while the VLM vision encoders use fp16.<br>

¹ The total used memory; LLM plus the VLM. <br> ² When an llm/vlm model is loaded for the first time from your disk to RAM or NPU, it is called a cold start.<br> The duration depends on your OS, I/O transfer rate, and memory mapping.<br> ³ Subsequent loading (warm start) takes advantage of the already mapped data in RAM. Mostly, only a few pointers need to be restored.<br><br> <img width="1000" height="700" alt="Plot_Tokens" src="https://github.com/user-attachments/assets/7342debe-769c-46c9-8a83-755caf7d67dc" /><br> <img width="1000" height="700" alt="PlotMemory" src="https://github.com/user-attachments/assets/cf4362e6-f644-46d3-9b74-d129b23d3c44" />

Example Usage

• see: https://github.com/Qengineering/Qwen3.5-9B-NPU

Notes

• KV Cache up to 16384
• This is not a Transformers-compatible model
• This repository provides precompiled NPU binaries
• CPU fallback is not supported