Downloads · 30 days
123
7% of all-time downloads
WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm
gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm is a image-text-to-text model from WaveCut. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for vllm. The card lists the license as apache-2.0.
compressed-tensors W4A16 quantization of WaveCut/gemma-4-26B-A4B-it-heretic for vLLM on a single 24 GB RTX 3090.
Downloads · 30 days
123
7% of all-time downloads
All-time downloads
1.8K
Public
Parameters
26.9B
17.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.1 GB · 100%
How the weights are stored.
I3224.5B · 91%
From the Hugging Face model README
compressed-tensors W4A16 quantization of
WaveCut/gemma-4-26B-A4B-it-heretic
for vLLM on a single 24 GB RTX 3090.
| Setting | Value |
|---|---|
| Method | RTN / memoryless_minmax |
| Weights | signed INT4, symmetric |
| Activations | BF16 at runtime |
| Group size | 64 |
| Format | compressed-tensors, pack-quantized |
| Source architecture | Gemma4ForConditionalGeneration |
| Source Heretic commit | e6d09d5c434a5e91326ce9d779786a91e7aaae35 |
| Original quantization stack | llm-compressor main, Transformers 5.8.0 |
Vision modules, routers, embeddings, and lm_head remain unquantized. Group
size 64 is required because Gemma 4 projection widths such as 2112 and 704 are
not divisible by 128. The exact recipe is stored in recipe.yaml.
On 2026-07-16, Google updated the Gemma 4 chat template but did not update the
model weights. The initial and current Google revisions have identical LFS
SHA256 values for both BF16 weight shards. This repository therefore keeps the
existing verified W4A16 tensors and refreshes the chat template, tokenizer
metadata, and processor metadata from Google revision
01e5b3ee840d3a9e0b0b493c593e85398a30ef75.
The production profile uses:
vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
--revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \
--served-model-name "Gemma 4 26B Heretic" \
--trust-remote-code \
--dtype bfloat16 \
--max-model-len 40192 \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.95 \
--generation-config vllm \
--no-enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4
The checkpoint was accepted for production on AI Farm with vLLM 0.25.1, Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0. On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case groups plus streaming checks. The matrix included exact needle recovery at 39,186 text tokens and 38,490 multimodal tokens, including concurrency 2. Both runtimes also passed 24/24 production-shaped Russian quality probes; median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1.
The full environment, timing table, harness, and raw JSONL artifacts are in
the AI Farm vLLM report.
The historical RTX 3090 quantization benchmark and harness remain available
under bench/ and scripts/.