Downloads · 30 days
8
8% of all-time downloads
whaoyang/gemma-3-4b-novision-rk3588-1.2.1
gemma-3-4b-novision-rk3588-1.2.1 is a text generation model from whaoyang. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization.
Downloads · 30 days
8
8% of all-time downloads
All-time downloads
100
Public
Repo size
5.3 GB
Likes
0
Public
Click a slice to open those files.
.rkllm5.3 GB · 99%
From the Hugging Face model README
This version of gemma-3-4b-novision has been converted to run on the RK3588 NPU using w8a8 quantization.
This model has been optimized with the following LoRA: NA
This model supports a max context length of 16384.
Compatible with RKLLM version: 1.2.1
rkllm parametersThis model runs well in limited testing with the following rkllm library paremeters:
n_keep = -1top_k = 64top_p = 0.95temperature = 0.7repeat_penalty = 1.0frequency_penalty = 1.0presence_penalty = 0.0mirostat = 0mirostat_tau = 5.0mirostat_eta = 0.1Pretty much anything by these folks: marty1885 and happyme531
Converted using https://github.com/c0zaut/ez-er-rkllm-toolkit
This model is a text-only version of google/gemma-3-4b-it, converted from the multimodal Gemma3ForConditionalGeneration architecture to the text-only Gemma3ForCausalLM architecture.
You can load and use this model the same way you would use the text-only google/gemma-3-1b-it version:
from transformers import AutoTokenizer, BitsAndBytesConfig, Gemma3ForCausalLM
import torch
model_id = "gghfez/gemma-3-4b-novision"
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
model = Gemma3ForCausalLM.from_pretrained(
model_id, quantization_config=quantization_config
).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
[
{
"role": "system",
"content": [{"type": "text", "text": "You are a helpful assistant."},]
},
{
"role": "user",
"content": [{"type": "text", "text": "Write a poem on Hugging Face, the company"},]
},
],
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device).to(torch.bfloat16)
with torch.inference_mode():
outputs = model.generate(**inputs, max_new_tokens=64)
outputs = tokenizer.batch_decode(outputs)