gemma-4-31b-it-int4-ov

gemma-4-31b-it-int4-ov

Gemma4-31B-it INT4 OpenVINO — OVMS 2026.3 Deployment (Legacy VLM Pipeline)

This repository contains google/gemma-4-31B-it (dense 31B) exported to OpenVINO IR format with INT4 weight compression, ready to serve via OpenVINO Model Server (OVMS) 2026.3 as an OpenAI-compatible endpoint.

Validated on: Intel Arc Pro B60 (24 GB), Ubuntu 24.04, OVMS 2026.3.0 native CLI binary.

Read this before you deploy: this model must be served with OVMS's legacy VLM pipeline, not VLM_CB (continuous batching / paged attention). VLM_CB produces corrupted output for this model — see Known Challenges for how this was isolated. This is not a quantization bug; the INT4 export itself is clean.

---

Model Files

Quantization parameters: INT4_ASYM, group_size=128, ratio=1.0 (no mixed precision).

---

Quick Start

Prerequisites

• OVMS 2026.3 native CLI binary (tarball, not a Docker image — see below)
• Intel GPU with Level Zero / OpenCL drivers installed
• At least 18 GB GPU VRAM (Arc Pro B60 or equivalent)
• /dev/dri render device accessible
• Python 3.12 with the pyovms extension on PYTHONPATH (bundled with the OVMS tarball)

Step 1 — Get OVMS 2026.3

mkdir -p ~/ovms_2026.3
wget https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/2026.3.0.39717a69b/ovms_ubuntu24_2026.3.0_python_on.tar.gz
tar -xzf ovms_ubuntu24_2026.3.0_python_on.tar.gz -C ~/ovms_2026.3
~/ovms_2026.3/ovms/bin/ovms --version

This is the standalone binary release, not a docker save archive — don't docker load it.

Step 2 — Download the model

mkdir -p ~/ovms_models
cd ~/ovms_models
pip install -U "huggingface_hub[hf_xet]" hf_xet
export HF_XET_HIGH_PERFORMANCE=1
export HF_XET_NUM_CONCURRENT_RANGE_GETS=32

hf download shailesh83/gemma-4-31b-it-int4-ov \
    --local-dir google/gemma-4-31B-it \
    --exclude "*.md" ".gitattributes"

Step 3 — Create config.json

cat > config.json << 'EOF'
{
    "model_config_list": [
        {
            "config": {
                "name": "google/gemma-4-31B-it",
                "base_path": "google/gemma-4-31B-it"
            }
        }
    ]
}
EOF

Step 4 — Start OVMS (native CLI, GPU)

export PYTHONPATH=~/ovms_2026.3/ovms/lib/python:$PYTHONPATH

~/ovms_2026.3/ovms/bin/ovms \
  --config_path ~/ovms_models/config.json \
  --rest_port 9001 \
  --allowed_local_media_path ~/ovms_models \
  --allowed_media_domains all \
  --log_level INFO

The PYTHONPATH export is required — without it, OVMS's Python-Mediapipe backend can't find its bundled pyovms extension module and the server will not come up.

CPU-only fallback: change device: "GPU" to device: "CPU" in graph.pbtxt, then drop --device/--group-add if you're also using Docker.

Step 5 — Wait for the model to load

curl -s http://localhost:9001/v3/models

Wait for google/gemma-4-31B-it to show up in the response.

---

Testing

Text-only inference

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain how a transformer attention mechanism works, step by step."}],
    "max_tokens": 400,
    "temperature": 0.1
  }'

Streaming

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain photosynthesis briefly."}],
    "max_tokens": 200,
    "stream": true
  }'

Automated benchmark + corruption check

A benchmark script is included in this repo (benchmark_gemma4.py) that runs a fixed set of prompts, reports tokens/sec, and flags known corruption signatures (useful as a regression check after any OVMS/optimum-intel upgrade):

python benchmark_gemma4.py \
  --url http://localhost:9001/v3/chat/completions \
  --model google/gemma-4-31B-it \
  --runs 3

---

Benchmark Results

Single Intel Arc Pro B60, legacy VLM pipeline, single-stream, temperature=0.1. 4 prompt types x 3 runs = 12 requests.

12/12 clean, 0 corrupted, 0 failed. Average 19.42 tok/s (range 18.84–19.66).

---

Known Challenges

This deployment went through a fairly involved debugging process. Summarized here so you don't have to repeat it:

1. INT4 corruption (red herring). Early attempts at exporting this model with INT4 compression (various group-size/ratio/symmetric combinations, across multiple optimum-intel branches) produced output that looked corrupted — garbled sub-tokens collapsing into repeated-token loops after roughly 100 tokens of otherwise coherent generation. This pattern held across CPU and GPU, across export methods, and across several quantization recipes, strongly suggesting an INT4-specific bug.
2. The actual root cause: OVMS's continuous-batching pipeline, not quantization. The same INT4 export produces clean output when tested directly via optimum-intel in Python and clean output when served through OVMS's legacy VLM pipeline — but corrupts when served through VLM_CB (continuous batching / paged attention), even with identical weights. This isolates the bug to VLM_CB's interaction with Gemma 4's hybrid attention head dimensions (256 sliding-window / 512 global) — likely a similar issue to one that required mainline vLLM to force a specific attention backend for this same model.
3. Gemma 4 is registered strictly as a VLM task, even for text-only use — exports must use --task image-text-to-text (direct optimum-cli) or --pipeline_type VLM/VLM_CB (export_model.py); text-generation-with-past is rejected outright.
4. Dependency conflicts are the biggest time sink. optimum-intel's Gemma 4 support currently lives on a fork (rkazants/optimum-intel@support_gemma_4), not mainline, and it has a narrow compatible window with transformers/safetensors/optimum-onnx versions. Installing OVMS's own export_model.py requirements.txt will silently install a different optimum-intel on top of the fork. Install requirements.txt first (full dependency resolution), then the fork second, unconstrained — this way the fork wins the conflict and pulls in compatible dependency versions.
5. OVMS's native GGUF loader does not support this model. If you already have a GGUF quant of Gemma 4 (e.g. from Unsloth), OVMS's built-in --pull --gguf_filename path will reject it — its recognized-architecture list includes gemma3 but not gemma4 as of this OVMS version. Use the IR export path in this repo instead.

---

Export Command (for reference)

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/common/export_models
pip install -r requirements.txt

pip install "git+https://github.com/rkazants/optimum-intel.git@support_gemma_4" \
    --extra-index-url https://download.pytorch.org/whl/cpu

python export_model.py text_generation \
    --source_model google/gemma-4-31B-it \
    --pipeline_type VLM \
    --weight-format int4 \
    --extra_quantization_params "--group-size 128 --ratio 1.0" \
    --target_device GPU \
    --model_repository_path ./ovms_models \
    --config_file_path ./ovms_models/config.json

Do not substitute --pipeline_type VLM_CB here — see Known Challenges.

---

Troubleshooting

---

Hardware Requirements

---

Known Limitations

• No continuous batching. The legacy VLM pipeline serves one request at a time; this configuration is validated for single-request serving only. Concurrent throughput is unvalidated.
• VLM_CB should not be used with this model until the underlying paged-attention issue with Gemma 4's hybrid head dimensions is fixed upstream in OpenVINO GenAI.
• Vision tower is present (SigLIP2) but unexercised by the text-only testing in this repo — image+text inference is architecturally supported but not validated here.

License

Gemma model weights are subject to the Gemma Terms of Use. This repository contains only the OpenVINO-converted weights with no modifications to model parameters.