gemma-4-E2B-it-int4-ov

gemma-4-E2B-it-int4-ov

google/gemma-4-E2B-it converted to OpenVINO IR with INT4 weight-only quantization (NNCF, group_size=128, asymmetric, ratio=1.0). Intended for serving with OpenVINO Model Server (OVMS) on CPU / Intel GPU.

OVMS compatibility note

This is a stitched single-file export: the split multimodal VLM IR (text-embeddings + per-layer-embeddings + language model) is merged into one openvino_model.xml (input_ids -> logits) that retains ScaledDotProductAttention, so OpenVINO Model Server's continuous-batching SDPA -> PagedAttention pipeline can serve it on GPU. The stock split/HF OpenVINO export fails to load in OVMS (No ScaledDotProductAttention operation observed); this one loads and serves.

Serve with OVMS

docker run -d --rm -p 8000:8000 -v $(pwd)/models:/models --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:2026.2.1-gpu \
  --rest_port 8000 --model_repository_path /models \
  --source_model Prog-up/gemma-4-E2B-it-int4-ov --target_device GPU --task text_generation

Then call the OpenAI-compatible endpoint at POST /v3/chat/completions.

Benchmark

~24.5 tok/s decode throughput (INT4, Intel iGPU, OVMS 2026.2.1, 256-token greedy decode).