gemma4-prometheus-gptq-4bit

gemma4-prometheus-gptq-4bit

<!-- polished-overview:start -->

Overview

gemma4-prometheus-gptq-4bit is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (5 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• quantize_config.json
• Additional configuration, tokenizer, processor, or shard files (14 visible artifacts total)

Quick start

vLLM (documented configuration)

vllm serve groxaxo/gemma4-prometheus-gptq-4bit \
  --quantization gptq_marlin \
  --dtype float16 \
  --trust-remote-code

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

GPTQ 4-bit quantized version of groxaxo/gemma4-prometheus-merged (Prometheus-steered google/gemma-4-31B-it).
Quantized with gptqmodel v5.8.0. 69.3% size reduction: 58 GiB → 17.9 GiB.

Related repositories

---

Quantization details

---

How to run

Requirements

• 1–2 × GPU with ≥ 20 GiB total VRAM (single 24 GB GPU works)
• gptqmodel >= 5.8.0 with the Gemma4 patch applied (see patches section)

Install

pip install gptqmodel>=5.8.0

Note: The standard gptqmodel package does not include Gemma4 support. Apply the patches from groxaxo/gemma4-prometheus-fixes before loading this model. The patches add Gemma4QModel and fix the alternating rotary-embedding shape mismatch.

Inference

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"

from gptqmodel import GPTQModel
from transformers import AutoTokenizer
import torch

model_id = "groxaxo/gemma4-prometheus-gptq-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GPTQModel.load(
    model_id,
    device_map="auto",          # single GPU or multi-GPU pipeline
    # max_memory={0: "22GiB"},  # uncomment to set per-GPU budget
)
model.eval()

messages = [{"role": "user", "content": "Explain gradient descent."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,       # suppress chain-of-thought tokens
)
ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    out = model.generate(
        ids,
        max_new_tokens=512,
        do_sample=False,
        temperature=None,
        top_p=None,
        pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Two-GPU pipeline parallel (2 × 24 GiB)

model = GPTQModel.load(
    model_id,
    device_map="balanced",
    max_memory={0: "22GiB", 1: "22GiB"},
)

---

Evaluation results

All tests run on 2 × RTX 3090 (24 GB each) in pipeline-parallel mode. True tensor-parallelism (TP=2) requires vLLM, which does not yet support the gemma4 architecture natively.

Coherence test (5/5 passed ✅)

Context length (2 × RTX 3090, GPTQ-4bit, no flash-attn)

Without flash-attn, the bottleneck is the attention matrix (O(n²) per layer), not KV cache storage. FP8 KV cache does not help here.

With flash-attn installed (estimated):

Recommendation: pip install flash-attn --no-build-isolation to unlock much longer contexts. The model supports up to 262 144 tokens.

Perplexity (WikiText-2)

ΔPPL = +1.9% is the meaningful signal. The absolute values are high because instruction-tuned models trained on chat data have poor raw-text likelihood.

KL divergence (this model vs merged reference)

KL ~4.77 nats is a typical result for 4-bit GPTQ on a 30B-class model. Part of the divergence is attributable to the bnb-8bit reference noise; true KL vs FP16 would be slightly lower.

---

Architecture notes (Gemma4 quirks)

Why layer_modules_strict = False: sliding-window attention layers omit v_proj, so a strict module check would fail. The flag allows partial matches.

Rotary embedding fix: Gemma4 alternates sliding_attention (head_dim=256) and full_attention (head_dim=512). gptqmodel cached the first layer's position_embeddings and replayed them for all layers, causing a shape mismatch at the first global-attention layer (layer 5). The fix regenerates position_embeddings per layer using the correct layer_type.

---

Patches required

See groxaxo/gemma4-prometheus-fixes for full patch diffs and instructions.

Required patches to gptqmodel:

1. gptqmodel/models/definitions/gemma4.py — new Gemma4QModel class
2. gptqmodel/models/auto.py — "gemma4" -> Gemma4QModel mapping
3. gptqmodel/looper/module_looper.py — free-memory device scheduling + per-layer rotary fix

---

Quantization config

{
  "bits": 4,
  "group_size": 128,
  "format": "gptq",
  "desc_act": false,
  "sym": true,
  "quant_method": "gptq"
}

---

Citation / acknowledgements

• Base model: google/gemma-4-31B-it
• Steering: Prometheus (local)
• Quantization: gptqmodel v5.8.0