gemma4-prometheus-merged

gemma4-prometheus-merged

<!-- polished-overview:start -->

Overview

gemma4-prometheus-merged is a model checkpoint packaged for compatible Hugging Face runtimes, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (2 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

Quick start

Getting started

Start with the upstream library named in the repository metadata and keep all configuration, tokenizer, processor, and weight files together. This repository is an artifact release, so the source project remains the authoritative reference for task-specific loading code.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

Prometheus-steered merged model — google/gemma-4-31B-it with Prometheus adversarial-steering baked in and all adapter weights merged into the base model.

Related repositories

---

What is this?

1. Downloaded google/gemma-4-31B-it (31 B parameters, BF16).
2. Ran Prometheus adversarial-steering optimization over 1 trial with 6 behaviors.
3. Merged the best steering vectors back into the base model weights.
4. Saved as a standalone, loadable model (no Prometheus runtime needed).

Size: ~58 GiB (two BF16 shards).

---

How to run

Minimum requirements

• 3 × RTX 3090 (24 GB each) or any combination totalling ≥ 65 GiB VRAM
• On 2 GPUs (≤ 48 GiB): load with BnB 8-bit (see below)

BF16 on 3 GPUs

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model_id = "groxaxo/gemma4-prometheus-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

messages = [{"role": "user", "content": "Explain gradient descent."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=512, do_sample=False,
                         pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

BnB 8-bit on 2 GPUs (48 GiB)

from transformers import AutoModelForImageTextToText, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "groxaxo/gemma4-prometheus-merged"

bnb_cfg = BitsAndBytesConfig(load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    quantization_config=bnb_cfg,
    device_map="auto",
    max_memory={0: "23GiB", 1: "23GiB"},
)
model.eval()

Important: disable thinking tokens

Always pass enable_thinking=False to apply_chat_template — otherwise the model may emit <start_of_turn>think tokens and spend tokens on chain-of-thought.

---

Evaluation results

All tests run on 2 × RTX 3090 (24 GB each) in pipeline-parallel mode (device_map="auto" via 🤗 Accelerate). True tensor-parallelism (TP=2) requires vLLM, which does not yet support the gemma4 architecture natively.

Coherence test (GPTQ-4bit model — same backbone)

5 diverse ML questions answered. 5/5 passed (≥ 15 words, on-topic).

Context length

Action: Installing flash-attn would increase usable context ~18×.

Perplexity (WikiText-2, sliding window stride=512, 4096 tokens)

Chat-tuned models have high raw-text perplexity. The ΔPPL between variants is the meaningful signal: +1.9% degradation from 4-bit quantization.

KL divergence (GPTQ-4bit vs merged reference)

Mean KL of ~4.77 nats reflects expected 4-bit quantization error relative to an 8-bit reference. Note: part of this KL is attributable to bnb-8bit noise in the reference; true KL vs FP16 merged would be somewhat lower.

---

Architecture notes

• 60 transformer layers alternating:
  • Sliding-window attention (window=1024, 16 KV heads, head_dim=256)
  • Full (global) attention (4 KV heads, head_dim=512)
• GQA with 32 query heads, 16/4 KV heads
• VLM wrapper (model.language_model) — vision tower present but text-only inference works

---

Patches applied

All source patches are documented at groxaxo/gemma4-prometheus-fixes.

Key fixes:

1. Prometheus PEFT adapter targeting — resolved exact module paths via named_modules() traversal to prevent over-matching vision layers.
2. Prometheus steering FP16 — defaulted steering vector compute dtype to FP16 (not FP32) to prevent VRAM OOM on quantized layers.
3. gptqmodel Gemma4 support — added Gemma4QModel definition with layer_modules_strict=False.
4. gptqmodel rotary embedding — per-layer position_embeddings regeneration with correct layer_type (sliding vs global).

---

Citation / acknowledgements

• Base model: google/gemma-4-31B-it
• Steering framework: Prometheus (local)
• Quantization: gptqmodel