gemma-4-E4B-it-W8A8-INT8-GPTQ

Gemma 4 E4B IT W8A8 INT8 GPTQ

This repository contains an INT8 W8A8 GPTQ quantized checkpoint derived from google/gemma-4-E4B-it.

The original Gemma 4 E4B IT model was developed by Google DeepMind and is released under the Apache 2.0 license. This quantized checkpoint keeps the same license metadata and links to Google's Gemma license page: https://ai.google.dev/gemma/docs/gemma_4_license

Quantization

• Base model: google/gemma-4-E4B-it
• Quantization tool: llmcompressor
• Modifier: GPTQModifier
• Scheme: W8A8
• Weight quantization: 8-bit integer, per-channel, symmetric
• Activation quantization: 8-bit integer, per-token, symmetric, dynamic
• actorder: null
• Calibration dataset: HuggingFaceH4/ultrachat_200k, split train_sft
• Calibration samples: 512
• Calibration max sequence length: 2048
• Random seed: 42

Multimodal encoders, lm_head, Gemma 4 AltUp/Laurel modules, and per-layer embedding projection paths were excluded from quantization and kept in BF16. The language-model attention and MLP Linear layers were quantized.

See recipe.yaml and config.json for the serialized quantization metadata.

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "lsm0729/gemma-4-E4B-it-W8A8-INT8-GPTQ"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM-based evaluation in this environment, the quantized checkpoint was tested with linear_backend=triton.

lm-eval run \
  --model vllm \
  --model_args 'pretrained=lsm0729/gemma-4-E4B-it-W8A8-INT8-GPTQ,dtype=auto,add_bos_token=false,max_model_len=15000,gpu_memory_utilization=0.5,enable_chunked_prefill=True,enforce_eager=True,trust_remote_code=True,linear_backend=triton' \
  --tasks mmlu_llama \
  --batch_size auto \
  --num_fewshot 5 \
  --apply_chat_template \
  --fewshot_as_multiturn

Evaluation Notes

The base model card reports Gemma 4 E4B MMLU Pro performance of 69.4%. This repository is a post-training quantized derivative; benchmark scores can differ from the original BF16 checkpoint depending on the harness, prompt template, thinking mode, backend, and runtime settings.

Provenance

• Original model: https://huggingface.co/google/gemma-4-E4B-it
• Original authors: Google DeepMind
• License: Apache 2.0
• License link: https://ai.google.dev/gemma/docs/gemma_4_license
• Quantized by: lsm0729