Qwen1.5-7B-Chat-rkllm

Hello, this is just a conversion of the Qwen 1.5 7B chat to rkllm, a model format that can run nativly on a rk3588/rk3576 chipset like an orange pi 5 (rk3588)

Also as far as the license is concerned, I just copied it from the orignal qwen file. it seems to be the llm format name, not a license name, I will update as needed.

How to run

Hardware I used an orange pi 5 Plus with 16 gb lpddr4 ram and a San Disk Extreme Pro 128GB UHS-1, C10, U3, V30, A2, 200MB/s Read, 90MB/s Write Card. The wifi card on the opi 5 I am using DID NOT WORK with this image, I hard wired it. I have the "5 Plus WiFi 6 Module Wireless Card 1201Mbps PCIe Wi-Fi6+BT5.2, Wireless Module R6" Realtek RTL8852BE-CG

I use the following tools and repos to get this started. Much apprication to Pelochus!

https://github.com/Pelochus/armbian-build-rknpu-0.9.6/releases A build of Armbian with the required 0.9.6 npu drivers installed.

https://github.com/Pelochus/ezrknn-llm A very easy way to install and build/make the rkllm runtime.

other models

https://huggingface.co/Pelochus

I used Balena Etcher to burn the downloaded image from https://github.com/Pelochus/armbian-build-rknpu-0.9.6/releases to an SD card and boot your Orange pi 5 Plus 16gb ddr4 version (only version I have tested)

Then run the following commands as a normal user from your home directory or working directory. The program will end up under the folder named ezrkknn-llm.

sudo apt update
sudo apt upgrade

sudo apt install cmake

git clone https://github.com/Pelochus/ezrknn-llm

cd ezrknn-llm
sudo bash install.sh

cd ezrknn-llm/rkllm-runtime/example/build/build_linux_aarch64_Release/

Use wget to download this repo to this directory

wget https://huggingface.co/kalabaddon/Qwen1.5-7B-Chat-rkllm/resolve/main/Qwen1-5-7B-Chat-rkllm.rkllm

./llm_demo Qwen1-5-7b-Chat-rkllm.rkllm

It took a WHILE to load ( at least from a fast sd card ), I was running this all from ssh and used a second connection running htop to monitor its loading progress, when the ram reaches ~9 gb you should expect this to be loaded. That means it seems like you need a 16gb version to run(minus when I was moving around some files, I already had the rkllm model on a thumb drive and used that to copy the qwen 1.5 7b file to the directory)

I have an orange pi pro 16gb ddr5 board in hand so will see if any images work on that. but otherwise good luck running this on the NPU, it loaded fine for me and seemed decent speed for using no gpu or cpu and only ram. (I feel it was like 1-2 tokens a sec, will upload video later maybe.)