Gemma-4-12B-it-W4A16

<img src="assert-logo-light.png" alt="ASSERT" height="26">

⚠️ Update — vLLM loading fixed

The first upload of this pack failed to load on vLLM with:

ValueError: There is no module or parameter named
'vision_embedder.patch_dense.weight' in Gemma4UnifiedForConditionalGeneration

Cause. llm-compressor recorded the ignore list in quantization_config using the module paths transformers exposes (model.embed_vision.patch_dense), while the saved tensors — and vLLM's own module tree — use vision_embedder.patch_dense. vLLM therefore found no ignore entry for that layer, built it as a quantized linear, and could not find packed weights for it.

Fix. The ignore entries are now regexes that match either naming. The weights are byte-identical — no re-quantization; only config.json changed. Re-download that one file, or re-pull the repo.

Verified: loads and serves on vLLM nightly, tensor-parallel 1, one RTX 3090, 274,181 tokens of KV cache.

gemma-4-12B-it — W4A16 (compressed-tensors)

Standard W4A16 quantization of google/gemma-4-12B-it, produced with llm-compressor (the official vLLM-team quantization toolkit) inside a reproducible Docker container. The artifact saves in compressed-tensors format and is drop-in loadable by vLLM — no upstream patches, no client-side shims; vLLM auto-detects the quantization config from the embedded config.json at load time.

This release is part of an ongoing series of vLLM-friendly quantized packs maintained by atlas, a self-evolving agent project run by Alex Adamopoulos at assert.gr.

Reproducibility

1 module of 328 fell back to round-to-nearest. GPTQ could not invert the Hessian there, so those weights were rounded without the error compensation the algorithm normally applies. They are correct, just less accurate than the rest. This is disclosed because the run exits 0 either way and nothing else would tell you.

License

Inherits the license of the base model. By using this artifact you agree to the original license at the source link above. Atlas / assert.gr adds no additional restrictions on the quantized weights.

Usage with vLLM

docker run --runtime=nvidia --gpus all \
    -p 8000:8000 \
    -e HF_TOKEN=hf_XXX \
    vllm/vllm-openai:latest \
    --model aleada/Gemma-4-12B-it-W4A16 \
    --limit-mm-per-prompt 'image=1' \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching

vLLM auto-detects compressed-tensors from the model's config — no --quantization flag required (it is accepted as a redundant hint). vLLM also picks the model's full native context window from config.json. If you hit KV-cache OOM on a smaller GPU, pin a shorter window with --max-model-len 16384 (or smaller) — leave it off to get the maximum the model was trained for. Once vLLM is running, hit it with any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="aleada/Gemma-4-12B-it-W4A16",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Hardware target

Requires CUDA compute-capability ≥ 8.0 (Ampere or newer). Verified on NVIDIA RTX 3090 (compute 8.6) where the W4A16 path runs the language tower at INT4 weights / BF16 activations through vLLM's compressed-tensors kernels. Vision encoder + multimodal projector remain BF16 by design — quantizing them gives negligible memory benefit relative to accuracy cost (matches the upstream llm-compressor multimodal-vision recommendation).

Weight-only INT4 is the point on this class of card: FP8 and NVFP4 checkpoints are native on Hopper and Blackwell but emulated or unusable on Ampere, where the INT4 Marlin kernels are what actually run fast.

Check this pack yourself

Quantization can drop or disable part of a model without failing: the pack loads, serves, and answers correctly while something its card says it kept is absent, or present and ignored by the runtime. Nothing errors, and the card still promises it.

Three tools read any published repo's metadata — safetensors headers and config.json, no weights — entirely in your browser, so each reads exactly what you could read yourself. Point them at this pack. Point them at someone else's.

Pack integrity check — whether the exclusion entries name real modules, whether anything from the source model failed to reach the pack, and whether anything is left at source precision without being declared.

Precision map — how much of a "4-bit" pack is actually 4-bit, and what stayed whole. Never all of it: embeddings, the output head and the norms are usually kept, so the honest figure is a fifth to two fifths of the bytes. It reads the dtypes rather than the tensor names, because five of the nine quantization toolchains store the packed payload under the plain name weight — a name-driven reader calls those packs full precision.

Reasoning-parser advisor — whether a model needs vLLM's --reasoning-parser and which one, read from its chat template. Getting this wrong is invisible: the wrong parser claims the entire output and content comes back empty, with no error anywhere.

About the maintainer

Alex Adamopoulos is the founder of assert.gr and the engineer behind the atlas self-evolving AI agent platform. Atlas runs a planner→executor→supervisor loop over a skill registry, backed by Postgres, Redis, Qdrant, and a multi-LLM vLLM deployment. Quantization releases like this one keep the open-source model ecosystem usable on consumer-grade hardware for self-hosted agent research.

Connect:

• HuggingFace: @aleada
• Company: assert.gr