Downloads · 30 days
76
16% of all-time downloads
jamescallander/deepseek-llm-7b-chat_w8a8_g128_rk3588.rkllm
deepseek-llm-7b-chat_w8a8_g128_rk3588.rkllm is a text generation model from jamescallander. Use it when you need the model to write or continue text. It is set up for rkllm. The card lists the license as other.
Author: @jamescallander Source model: deepseek-ai/deepseek-llm-7b-chat
Downloads · 30 days
76
16% of all-time downloads
All-time downloads
472
Public
Repo size
7.9 GB
Likes
0
Public
Click a slice to open those files.
.rkllm7.9 GB · 100%
From the Hugging Face model README
Author: @jamescallander
Source model: deepseek-ai/deepseek-llm-7b-chat
Target: Rockchip RK3588 NPU via RKNN-LLM Runtime
This repository hosts a conversion of
DeepSeek-LLM-7B-Chatfor use on Rockchip RK3588 equipped single board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using rknn-llm toolkit.
.rkllm artifactThis build is intended for experimentation and deployment of DeepSeek-LLM-7B-Chat on Rockchip RK3588-based SBCs.
The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from airockchip's github page.
Download and install the required packages as per the toolkit's instructions.
The simplest way the deploy the .rkllm converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo
python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
--rkllm_model_path <MODEL_PATH>/rk3588-deepseek-llm-7b-chat.rkllm \
--target_platform rk3588
A basic format for message request is:
{
"model":"deepseek-7b",
"messages":[{
"role":"user",
"content":"<YOUR_PROMPT_HERE>"}],
"stream":false
}
Example request using curl:
curl -s -X POST <MODEL_SERVER_IP_ADDRESS>:8080/rkllm_chat \
-H 'Content-Type: application/json' \
-d '{"model":"deepseek-7b","messages":[{"role":"user","content":"In one sentence, who was Napoleon?"}],"stream":false}'
The response is formated in the following way:
{
"choices":[{
"finish_reason":"stop",
"index":0,
"logprobs":null,
"message":{
"content":"<MODEL_REPLY_HERE>",
"role":"assistant"}}],
"created":null,
"id":"rkllm_chat",
"object":"rkllm_chat",
"usage":{
"completion_tokens":null,
"prompt_tokens":null,
"total_tokens":null}
}
Example response:
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Napoleon Bonaparte (1769-1821) was a French military leader and statesman who rose to power during the French Revolution, becoming Emperor of France from 1804 to 1815 and implementing various reforms and conquests that had a lasting impact on European history.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
This server exposes an OpenAI-compatible Chat Completions API.
You can connect it to any OpenAI-compatible client or UI (for example: Open WebUI)
http://<SERVER_IP_ADDRESS>:8080 and use the endpoint: /rkllm_chatmodel field matches the converted model’s name, for example:{
"model": "DeepSeek-LLM-7B-Chat",
"messages": [{"role":"user","content":"Hello!"}],
"stream": false
}
This conversion follows the license of the source model: DeepSeek LLM license.
.rkllm format for RK3588 SBCs.For more information on the deployment and use of .rkllm models on RK3588 platforms, please refer to the RKNN-LLM toolkit documentation.