Downloads · 30 days
5
11% of all-time downloads
whaoyang/gemma-3-4b-novision-quant-rk3588-1.2.1
gemma-3-4b-novision-quant-rk3588-1.2.1 is a text generation model from whaoyang. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization and with the quantdataset.json file from this repo as the value for the dataset param of the rkllm.build() command.
Downloads · 30 days
5
11% of all-time downloads
All-time downloads
45
Public
Repo size
5.3 GB
Likes
0
Public
Click a slice to open those files.
.rkllm5.3 GB · 99%
From the Hugging Face model README
This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization and with the quant_dataset.json file from this repo as the value for the dataset param of the rkllm.build() command.
This converted rkllm model differs from my other gemma-3-4b-novision-rk3588-1.2.1 model in that rkllm.build() was run with param dataset=quant_dataset.json instead of param dataset=None in the other model.
The quant_dataset.json file was generated with Rockchip's generate_data_quant.py script with the following arguments:
This model has been optimized with the following LoRA: NA
This model supports a max context length of 16384.
Compatible with RKLLM version: 1.2.1
rkllm parametersThis model runs well in limited testing with the following rkllm library paremeters:
n_keep = -1top_k = 64top_p = 0.95temperature = 0.7repeat_penalty = 1.0frequency_penalty = 1.0presence_penalty = 0.0mirostat = 0mirostat_tau = 5.0mirostat_eta = 0.1It is recommended to also apply a specific chat template using the following Python methods(that hook the rkllm library):
# System prompt taken from https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune#official-recommended-inference-settings
system_prompt = "<bos><start_of_turn>user\nHello!<end_of_turn>\n<start_of_turn>model\nHey there!<end_of_turn>\n<start_of_turn>user\nWhat is 1+1?<end_of_turn>\n<start_of_turn>model\n"
prompt_prefix = "<start_of_turn>user\n"
prompt_postfix = "<end_of_turn>\n<start_of_turn>model\n"
rkllm_lib.rkllm_set_chat_template(
llm_handle,
ctypes.c_char_p(system_prompt.encode('utf-8')),
ctypes.c_char_p(prompt_prefix.encode('utf-8')),
ctypes.c_char_p(prompt_postfix.encode('utf-8')))
Pretty much anything by these folks: marty1885 and happyme531
Converted using https://github.com/c0zaut/ez-er-rkllm-toolkit