gemma4-prometheus-fixes

Gemma4 Prometheus fixes

<!-- polished-overview:start -->

Overview

gemma4-prometheus-fixes is a development repository associated with the model-building workflow, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• FIXES.md

Quick start

Getting started

Start with the upstream library named in the repository metadata and keep all configuration, tokenizer, processor, and weight files together. This repository is an artifact release, so the source project remains the authoritative reference for task-specific loading code.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

This repo documents the local patches that made the Gemma4 + Prometheus + GPTQ pipeline work on this machine.

What was fixed

• Prometheus adapter targeting now resolves exact module paths instead of suffix matches.
• Prometheus steering now dequantizes to FP16 on CUDA by default to avoid VRAM blowups.
• GPTQModel now recognizes gemma4, uses a Gemma4 module tree, tolerates missing v_proj, and refreshes rotary position_embeddings per layer.
• Quantization scheduling is free-memory-aware and does not rely on naive round-robin GPU assignment.

Related repos

• Workflow and reproduction scripts: groxaxo/gemma4-prometheus-workflow
• Merged model: groxaxo/gemma4-prometheus-merged
• GPTQ model: groxaxo/gemma4-prometheus-gptq-4bit

Reproduce

Use the workflow repo with the same conda env described there. The workflow README includes the exact commands used for:

1. Prometheus text-only inference
2. merged-model export
3. GPTQ quantization