Downloads · 30 days
3
0% of all-time downloads
Echo9Zulu/gemma-3-4b-it-qat-int4_asym-ov
gemma-3-4b-it-qat-int4_asym-ov is a image-text-to-text model from Echo9Zulu. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
My Project OpenArc, an inference engine for OpenVINO, now supports this model and serves inference over OpenAI compatible endpoints for text to text and text with vision!
Downloads · 30 days
3
0% of all-time downloads
All-time downloads
666
Public
Repo size
3.5 GB
Likes
3
Public
Click a slice to open those files.
.bin3.5 GB · 99%
From the Hugging Face model README
My Project OpenArc, an inference engine for OpenVINO, now supports this model and serves inference over OpenAI compatible endpoints for text to text and text with vision!
We have a growing Discord community of others interested in using Intel for AI/ML.
This model was converted to the OpenVINO IR format using the following Optimum-CLI command:
optimum-cli export openvino -m ""input-model"" --task image-text-to-text --weight-format int8 ""converted-model""
Well, it demonstrates how to inference in python and what parts of that code are important for benchmarking performance. Text generation offers different challenges than text-generation with images; for examples, vision encoders often use different strategies for handling properties an image can have; to get good performance be mindful of image resolution. In practice this can translate to higher memory usage, reduced throughput and greater variety in results. Gemma-3 uses SigLIP 2 which has many SOTA optimizations; even so, effort in the preprocessing stage of a pipeline makes a world of difference.
To run the test code:
pip install optimum[openvino]+https://github.com/huggingface/optimum-intel
import time
from PIL import Image
from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM
model_id = "Echo9Zulu/gemma-3-4b-it-int8_asym-ov" # Can be an HF id or a path
ov_config = {"PERFORMANCE_HINT": "LATENCY"} # Optimizes for first token latency and locks to single CPU socket
print("Loading model... this should get faster after the first generation due to caching behavior.")
print("")
start_load_time = time.time()
model = OVModelForVisualCausalLM.from_pretrained(model_id, export=False, device="CPU", ov_config=ov_config) # For GPU use "GPU.0"
processor = AutoProcessor.from_pretrained(model_id) # Instead of using AutoTokenizers we use AutoProcessor which routes to the appropriate input processor i.e, how does a model expect image tokens.
# Under the hood this takes care of model specific preprocessing and has functionality overlap with AutoTokenizers.
end_load_time = time.time()
image_path = r"" # This script expects .png
image = Image.open(image_path)
image = image.convert("RGB") # Required by gemma3. In practice this would need to be handled at the engine level OR in model-specifc pre-processing.
conversation = [
{
"role": "user",
"content": [
{
"type": "image"
},
{"type": "text", "text": "Describe this image."},
],
}
]
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")
input_token_count = len(inputs.input_ids[0])
print(f"Sum of image and text tokens: {len(inputs.input_ids[0])}")
start_time = time.time()
output_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids = [output_ids[len(input_ids) :] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
num_tokens_generated = len(generated_ids[0])
load_time = end_load_time - start_load_time
generation_time = time.time() - start_time
tokens_per_second = num_tokens_generated / generation_time
average_token_latency = generation_time / num_tokens_generated
print("\nPerformance Report:")
print("-"*50)
print(f"Input Tokens : {input_token_count:>9}")
print(f"Generated Tokens : {num_tokens_generated:>9}")
print(f"Model Load Time : {load_time:>9.2f} sec")
print(f"Generation Time : {generation_time:>9.2f} sec")
print(f"Throughput : {tokens_per_second:>9.2f} t/s")
print(f"Avg Latency/Token : {average_token_latency:>9.3f} sec")
print(output_text)