Downloads · 30 days
22
18% of all-time downloads
jamescallander/MediPhi-PubMed_w8a8_g128_rk3588.rkllm
MediPhi-PubMed_w8a8_g128_rk3588.rkllm is a text generation model from jamescallander. Use it when you need the model to write or continue text. It is set up for rkllm. The card lists the license as mit.
Author: @jamescallander Source model: microsoft/MediPhi-PubMed · Hugging Face
Downloads · 30 days
22
18% of all-time downloads
All-time downloads
125
Public
Repo size
4.3 GB
Likes
1
Public
Click a slice to open those files.
.rkllm4.3 GB · 100%
From the Hugging Face model README
Author: @jamescallander
Source model: microsoft/MediPhi-PubMed · Hugging Face
Target: Rockchip RK3588 NPU via RKNN-LLM Runtime
This repository hosts a conversion of
MediPhi-PubMedfor use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the RKNN-LLM toolkit
w8a8_g128.rkllm artifact.rkllm format for RK3588 SBCs🛑 This model is not a substitute for professional medical advice, diagnosis, or treatment.
w8a8_g128) may further reduce output fidelity.The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from airockchip's github page.
Download and install the required packages as per the toolkit's instructions.
The simplest way the deploy the .rkllm converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo
python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
--rkllm_model_path <MODEL_PATH>/MediPhi-PubMed_w8a8_g128_rk3588.rkllm \
--target_platform rk3588
A basic format for message request is:
{
"model":"MediPhi-PubMed",
"messages":[{
"role":"user",
"content":"<YOUR_PROMPT_HERE>"}],
"stream":false
}
Example request using curl:
curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
-H 'Content-Type: application/json' \
-d '{"model":"MediPhi-PubMed","messages":[{"role":"user","content":"Explain what a retrovirus is in 2 or 3 sentences."}],"stream":false}'
The response is formated in the following way:
{
"choices":[{
"finish_reason":"stop",
"index":0,
"logprobs":null,
"message":{
"content":"<MODEL_REPLY_HERE">,
"role":"assistant"}}],
"created":null,
"id":"rkllm_chat",
"object":"rkllm_chat",
"usage":{
"completion_tokens":null,
"prompt_tokens":null,
"total_tokens":null}
}
Example response:
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"A retrovirus is a type of virus that uses RNA as its genetic material and reverse transcribes this RNA into DNA, which then integrates into the host cell's genome. This process allows for viral replication within infected cells. Retroviruses are notable because they can cause chronic infections due to their ability to integrate with a host’s genomic material and evade immune responses effectively.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
This server exposes an OpenAI-compatible Chat Completions API.
You can connect it to any OpenAI-compatible client or UI (for example: Open WebUI)
http://<SERVER_IP_ADDRESS>:8080 and use the endpoint: /rkllm_chatmodel field matches the converted model’s name, for example:{
"model": "MediPhi-PubMed",
"messages": [{"role":"user","content":"Hello!"}],
"stream": false
}
This conversion follows the license of the source model: markdown/mit.md · choosealicense/licenses at main
NOTICE