Gemma-3-12B-it-W4A16

<img src="assert-logo-light.png" alt="ASSERT" height="26">

⚠️ Update (2026-07-05) — vLLM loading fixed

An earlier revision of these weights failed to load on vLLM (≤ 0.23.x), raising:

KeyError: embeddings.patch_embedding.bias

Cause. transformers > 4.52.2 changed the multimodal weight-naming convention (to model.vision_tower.* / model.language_model.*), and GPTQModel followed it during quantization — but vLLM still expects the previous naming (vision_tower.vision_model.* / language_model.model.*), so the SigLIP vision tower failed to map (upstream: llm-compressor #1546).

Fix. The tensor keys are remapped to the vLLM-expected convention. The quantized weights are byte-identical — no re-quantization, nothing else changed. Just re-download the safetensors (or re-pull the repo) and it loads.

Verified: loads + runs on vLLM v0.23.0 (TP = 2), and still loads under transformers. Details in the Community tab.

gemma-3-12b-it-qat-q4_0-unquantized — W4A16 (GPTQModel)

Standard W4A16 GPTQ quantization of Lightricks/gemma-3-12b-it-qat-q4_0-unquantized, produced with GPTQModel 7.1.0 inside a reproducible Docker container. The artifact is drop-in loadable by vLLM (≤ 0.23.x, tensor-parallel) — vLLM auto-detects the GPTQ method from the embedded quantization_config at load time. (See the update note above: the vision-tower key naming is now aligned to vLLM's expectation.)

Produced with GPTQModel rather than the llm-compressor path used for the rest of this collection. The source is Google's quantization-aware-trained release restored to BF16 by Lightricks, so the weights entering GPTQ have already been trained to tolerate 4-bit rounding — the vision tower and multimodal projector still stay BF16, as they do across this collection.

Maintained by Alex Adamopoulos at assert.gr as part of an ongoing series of vLLM-friendly quantized packs targeting underserved 2026 models.

Reproducibility

GPTQ inverts a Hessian per module, and too little damping makes that inversion fail. GPTQModel starts at a higher damping than most tooling and raises it automatically on failure, so no module here silently degraded to plain round-to-nearest.

What the quantization cost

A quantized pack is a trade, and this is the side of it you can check before spending the download. Quantization does not damage a model evenly: what survives is the kind of work where a small numeric shift changes nothing, and what suffers is the kind where it changes the answer. Which of those your own work is decides whether this pack is right for you, and the rows below are how you tell — read the generative row if you run arithmetic, tools or long chains of reasoning, and the multiple-choice rows if you classify, rank or chat.

Both arms were run identically — same tasks, same harness, same machine — against Lightricks/gemma-3-12b-it-qat-q4_0-unquantized. Recovery is this pack's score over the source model's.

A ratio needs both halves. An absolute score for a quantized pack cannot say what the quantization cost, because the number it should be compared against is missing — and the source's own published figures are not that number either, since they were produced on other hardware with other settings. So the source model was scored here, in the same session as the pack, and both columns below come from that one run.

• Generative tasks: 97.2–97.5 % — the ones that write an answer and are graded on it.
• Multiple-choice tasks: 97.5–100.3 % — ranking fixed options, where rounding error rarely changes the winner.

There is no single figure here on purpose. The two kinds of task answer different questions, and averaging them hides the answer that matters: on one of our packs the mean came to 100.1% while the generative task had lost nearly nine points. Multiple-choice scoring asks the model to rank options it is given — error that ruins a calculation rarely reverses such a ranking, and a row above 100% is noise, not improvement. A generated answer graded on its final number is where 4-bit rounding compounds.

Measured with lm_eval 0.4.13, torch 2.12.0, transformers 5.16.1, full standard task configurations, each prompt bounded to 4096 tokens, on 2026-09-08T13:54:52Z.

These tasks are text-only. This pack is multimodal and its vision tower is excluded from quantization, so nothing here measures the image path.

License

Inherits the license of the base model. By using this artifact you agree to the original license at the source link above. Atlas / assert.gr adds no additional restrictions on the quantized weights.

Usage with vLLM

docker run --runtime=nvidia --gpus all \
    -p 8000:8000 \
    -e HF_TOKEN=hf_XXX \
    vllm/vllm-openai:latest \
    --model aleada/Gemma-3-12B-it-W4A16 \
    --limit-mm-per-prompt 'image=1' \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching

vLLM auto-detects the GPTQ format from the embedded quantization_config.quant_method. Passing --quantization gptq is allowed but redundant. vLLM also picks the model's full native context window from config.json. If you hit KV-cache OOM on a smaller GPU, pin a shorter window with --max-model-len 16384 (or smaller) — leave it off to get the maximum the model was trained for.

Vision encoder + multimodal projector remain BF16 by design — quantizing them gives negligible memory benefit relative to accuracy cost. Only the language tower's Linear weights are W4 GPTQ-quantized. Once vLLM is running, hit it with any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="aleada/Gemma-3-12B-it-W4A16",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Hardware target

Requires CUDA compute-capability ≥ 7.5 (Turing+). Verified on NVIDIA RTX 3090 (compute 8.6) where the W4A16 path runs the quantized Linears at INT4 weights / BF16 activations.

Check this pack yourself

Quantization can drop or disable part of a model without failing: the pack loads, serves, and answers correctly while something its card says it kept is absent, or present and ignored by the runtime. Nothing errors, and the card still promises it.

Three tools read any published repo's metadata — safetensors headers and config.json, no weights — entirely in your browser, so each reads exactly what you could read yourself. Point them at this pack. Point them at someone else's.

Pack integrity check — whether the exclusion entries name real modules, whether anything from the source model failed to reach the pack, and whether anything is left at source precision without being declared.

Precision map — how much of a "4-bit" pack is actually 4-bit, and what stayed whole. Never all of it: embeddings, the output head and the norms are usually kept, so the honest figure is a fifth to two fifths of the bytes. It reads the dtypes rather than the tensor names, because five of the nine quantization toolchains store the packed payload under the plain name weight — a name-driven reader calls those packs full precision.

Reasoning-parser advisor — whether a model needs vLLM's --reasoning-parser and which one, read from its chat template. Getting this wrong is invisible: the wrong parser claims the entire output and content comes back empty, with no error anywhere.

About the maintainer

Alex Adamopoulos is the founder of assert.gr and the engineer behind the atlas self-evolving AI agent platform.

Connect:

• HuggingFace: @aleada
• Company: assert.gr