Gemma4-31B-Int8-RBLN-0.11.2

iAcloud/Gemma4-31B-Int8-RBLN-0.11.2

Gemma4-31B-Int8-RBLN-0.11.2 is a image-text-to-text model from iAcloud. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for optimum-rbln. The card lists the license as gemma.

google/gemma-4-31B-it (multimodal, image-text-to-text), using the INT8 checkpoint from lokeshe09/gemma-4-31B-it-INT8, pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln and ready to serve with vllm-rbln.

Downloads · 30 days

100% of all-time downloads

All-time downloads

Public

Repo size

54 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.rbln48.2 GB · 89%

At a glance

Task: Image-Text-to-Text
Library: optimum-rbln
License: gemma
Model type: gemma4
Access: Public
Created: Sep 7, 2026
Updated: Sep 7, 2026
SHA: f108d131

Try a prompt

Base models

Model card

From the Hugging Face model README

Gemma4-31B-Int8-RBLN-0.11.2

google/gemma-4-31B-it (multimodal, image-text-to-text), using the INT8 checkpoint from lokeshe09/gemma-4-31B-it-INT8, pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln and ready to serve with vllm-rbln.

This repository contains the compiled artifact only. RBLN NPUs and the RBLN SDK are required to run it.

Hardware and software

Component	Configuration
NPU	16× RBLN-CA25 (language model: 16 devices; vision tower and multimodal projector: device 0, shared)
Compiler	`rebel-compiler==0.11.2`
Runtime	`optimum-rbln==0.11.2`, `vllm-rbln==0.11.2.post1`, `vllm==0.24.0`, `transformers==5.8.1`
Quantization	INT8 weights / FP16 activations (W8A16, weight-only)

Compile configuration

Option	Value
`max_seq_len`	262,144
`batch_size`	8
`decoder_batch_sizes`	`[1, 2, 4, 8]`
`attn_impl` / `kvcache_partition_len`	`flash_attn` / 16,384
`cache_impl` / `sliding_window`	`hybrid` / 1,024
`prefill_chunk_size`	128
`kvcache_num_blocks`	36 (589,824 shared KV tokens)
Vision `max_soft_tokens` / image prefill	280 / `image_prefill_384`

The 36 KV blocks were selected by the SDK estimator with memory_budget=0.9. The shared KV capacity is approximately 2.25 full 262K contexts; shorter requests can use the compiled dynamic batches up to batch 8.

Serving

export VLLM_PLUGINS=rbln
export VLLM_RBLN_USE_VLLM_MODEL=0
export VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK=16
export VLLM_RBLN_SUB_BLOCK_CACHE=false

vllm serve <this-repo-or-local-path> \
  --dtype float32 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 128 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Notes:

Do not pass --tensor-parallel-size; the 16-device split is baked into the compiled language model and vllm-rbln runs it with TP=1.
Keep --max-num-seqs 8 and --max-num-batched-tokens 128 aligned with the compiled decoder and prefill graphs.
Prefix caching must remain disabled. Gemma4 uses multi-group hybrid KV caches, which sub-block prefix caching does not support in this SDK version.
Thinking is controlled per request with chat_template_kwargs={"enable_thinking": true}. Parsed reasoning is returned separately from the final response content.
The artifact accepts one image per prompt with the compiled vision profile.

License

Derived from google/gemma-4-31B-it. Use is subject to the Gemma Terms of Use.

Task: Image-Text-to-Text
Library: optimum-rbln
Type: gemma4
License: gemma
Created: Sep 7, 2026
Updated: Sep 7, 2026