Downloads · 30 days
32
13% of all-time downloads
useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision
Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision is a image-text-to-text model from useful-quants. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for vllm. The card lists the license as apache-2.0.
This is a calibrated W4A16 (INT4 weight-only) quantization of Mitchins/gemma-4-26B-A4B-it-qat-q40-unquantized-uncensored-heretic, a QAT/Heretic tuned Gemma 4 26B-A4B instruction checkpoint. It is not a stock, bit-iden…
Downloads · 30 days
32
13% of all-time downloads
All-time downloads
252
Public
Parameters
25.8B
15.7 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors15.7 GB · 100%
How the weights are stored.
I3224.5B · 95%
From the Hugging Face model README
This is a calibrated W4A16 (INT4 weight-only) quantization of Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic, a QAT/Heretic tuned Gemma 4 26B-A4B instruction checkpoint. It is not a stock, bit-identical conversion of the ordinary public Google IT checkpoint. The model remains a Gemma 4 26B-A4B instruction/multimodal model; useful-quants performed the W4A16 conversion and packaging, not the original model development or the parent tuning.
The language-model linear and MoE expert weights are packed with compressed-tensors W4A16/G64; activations remain BF16. The vision tower, routers, embeddings, and LM head remain BF16/unquantized. The packed payload is 14.58 GiB (model.safetensors); the full repository is approximately 14.61 GiB including tokenizer and processor files.
Text generation and real-image inference were validated on one RTX 3090 24 GB with vLLM, Ampere Marlin dense/MoE kernels, and CUDA graphs. A 131,072-token total sequence window (128K) was also validated on that reference system.
G64 is deliberate: the relevant Gemma 4 26B-A4B projection dimensions (intermediate_size=2112, moe_intermediate_size=704) divide cleanly by 64 but not by 128.
| Property | Value |
|---|---|
| Architecture | Gemma 4 26B-A4B MoE instruction/multimodal |
| Immediate source | Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic at d62fa836581401beafd08fdd624927e31dcac6ee |
| Quantization | GPTQ-style W4A16, symmetric INT4, group size 64 |
| Runtime format | compressed-tensors pack-quantized |
| Precision retained | BF16 activations, vision tower, routers, embeddings, LM head |
| Primary runtime | vLLM (validated on a pinned main commit; see Runtime compatibility) |
| Reference GPU | RTX 3090 24 GB (Ampere) |
| Validated operational window | 131,072 total tokens, including generated tokens |
| Architectural context limit | 262,144 tokens |
| License | Apache-2.0, inherited from the Gemma parent |
The immediate parent is Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic, revision d62fa836581401beafd08fdd624927e31dcac6ee. That parent is an uncensored/abliterated Heretic derivative of the QAT-trained google/gemma-4-26B-A4B-it-qat-q4_0-unquantized lineage, itself based on Google Gemma 4 26B-A4B-IT. The local export was frozen before conversion, including source-file SHA256 hashes, tokenizer, processor, chat template, generation configuration, and architecture configuration. See provenance/calibration-and-source.json.
The parent model's Apache-2.0 license and Gemma 4 license link apply. This quantization does not change the parent model's intended use, safety guidance, or license terms.
The saved quantization_config is the source of truth:
Linear language-model and MoE expert weights: 4-bit signed integer weights, W4A16, group-wise G64, symmetric, static activation ordering, memoryless_minmax observer, pack-quantized storage.lm_head tensor; output weights are tied through the BF16 token embedding.The GPTQ modifier also used a 128-token block size and dampening_frac=0.01; the complete public-safe recipe is in provenance/calibration-and-source.json.
Calibration was multimodal rather than text-only so the conversion observed both normal language traffic and the image-processing/template path.
20260812.
Salesforce/wikitext, wikitext-103-raw-v1 train.HuggingFaceTB/smoltalk, openhermes-100k.HuggingFaceTB/smoltalk, apigen-80k.The exact datasets, sample counts, seed, selection method, image hash-manifest digest, exclusions, and software versions are in provenance/calibration-and-source.json. Calibration media are deliberately not included in this release.
| Model | Perplexity | Delta vs. BF16 |
|---|---|---|
| Tuned QAT/Heretic BF16 parent | 388.150 | — |
| W4A16-G64 | 432.318 | +44.168 / +11.38% |
The result uses held-out Salesforce/wikitext wikitext-2-raw-v1 test text, which was not used for calibration. Both rows use the same frozen tokenizer, four contiguous non-empty 512-token test-corpus windows (2,044 scored next-token positions), next-token NLL arithmetic, and vLLM prompt_logprobs=1 scorer. BF16 is executed with vLLM CPU offload because the parent checkpoint exceeds a 24 GB GPU. See eval/evaluate_gemma4_ppl_vllm.py and benchmarks/perplexity.json.
This is a compact quantization sanity evaluation, not a replacement for task-specific evaluation of the tuned parent.
The final artifact was fresh-reloaded and passed:
For images, use the Gemma 4 processor or OpenAI-compatible multimodal request format. The source chat template emits Gemma's actual <|image|> placeholder semantics; do not manually write a raw <image> token into a text prompt.
Gemma 4 supports video in its broader architecture, but video was neither required nor validated for this image-focused release profile. Unrestricted renderer warmup needlessly exercises a video path, so the validated image-serving profile explicitly uses:
--limit-mm-per-prompt '{"image":1,"video":0}'
This is a runtime warmup/configuration consideration, not a text or image generation failure.
Peak observed VRAM in the full-checkpoint performance test was 23,914 MiB; the benchmark conditions are below.
Reference setup: physical CUDA:1 RTX 3090, vLLM main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6, CUDA graphs enabled, max_model_len=1024, max_num_seqs=32, max_num_batched_tokens=8192, prefix caching disabled, three repetitions per point, and fixed 128-token greedy ignore_eos outputs. Values are aggregate verified output tokens/s, not per-request rate.
| Active requests | Text aggregate output tok/s | One image/request aggregate output tok/s |
|---|---|---|
| 1 | 156.6 | 127.9 |
| 4 | 486.9 | 262.6 |
| 8 | 820.4 | 429.9 |
| 16 | 1,269.7 | 518.5 |
| 32 | 1,948.3 | 461.6 |
The text test is decode-dominant. The one-image/request test includes image processing and multimodal prefill, so it is not directly comparable to the text-only decode number. B32 maximized measured text aggregate throughput. B16 is the measured image-workload knee; B32 was slower for that workload because it exceeds the efficient 8,192-token multimodal batching envelope. vLLM profiling allocated encoder cache for up to 29 maximum-size images in this configuration. Raw public-safe measurements: benchmarks/rtx3090-performance.json.
Gemma 4 advertises an architectural maximum of 262,144 tokens. That is not the same as what this 24 GB reference deployment can serve.
The validated RTX 3090 operational maximum is 131,072 total tokens (128K). A near-limit request with 130,816 input tokens plus 64 generated tokens completed with CUDA graphs, Marlin W4A16 dense/MoE paths, no OOM, and no allocator warning.
6444425401 bytes.135083 tokens.4011 tokens (about 3%).max_model_len includes generation: input_tokens + max_tokens <= 131072. For example, with a 2,048-token generation budget, the maximum input is 129,024 tokens. The exact fixed reservation is hardware- and runtime-specific; do not assume every 24 GB GPU has the same headroom.
This is the validated 128K RTX 3090 reference profile:
CUDA_VISIBLE_DEVICES=1 \
vllm serve useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision \
--quantization compressed-tensors \
--dtype bfloat16 \
--max-model-len 131072 \
--kv-cache-memory-bytes 6444425401 \
--max-num-seqs 17 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--enable-prefix-caching \
--scheduling-policy priority \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--host 0.0.0.0 \
--port 8000
Do not combine this exact fixed-KV profile with --gpu-memory-utilization: explicit cache bytes are authoritative. The byte value is a validated RTX 3090 reference setting, not a universal default.
Admission policy matters as much as priority:
max_num_seqs=17.priority: 100.priority: -10.Example OpenAI-compatible high-priority request, reserving a 2K response budget:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision",
"messages": [{"role": "user", "content": "Summarize the supplied context."}],
"max_tokens": 2048,
"temperature": 0.2,
"priority": -10
}'
For independent text-only jobs, use up to 32 active requests. The measured B32 point reached 1,948.3 aggregate verified output tok/s under the benchmark conditions above. This is a throughput profile, not the long-context mixed-service reservation.
For one image per request, B16 is the measured knee (518.5 aggregate output tok/s). Retain the 8,192-token batching envelope and disable video unless it is explicitly required:
--max-num-seqs 16 --max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image":1,"video":0}'
Known-good final validation environment:
| Component | Version |
|---|---|
| vLLM | main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6 (0.1.dev1+g7ca49fbe4) |
| Transformers | 5.15.1 |
| PyTorch | 2.13.0+cu130 |
| CUDA runtime | 13.0 |
| compressed-tensors runtime | 0.17.0 |
| Conversion artifact metadata | compressed-tensors 0.18.0 |
vLLM 0.27.1 did not contain the necessary handling for Transformers-5.15-era heterogeneous Gemma 4 attention metadata. The artifact was not the cause. Use the pinned commit above or a later vLLM release that explicitly includes the corresponding Gemma 4 heterogeneous-config support. Transformers-native inference is not claimed as validated for this packed artifact; vLLM is the supported runtime documented here.
Use an OpenAI-compatible multimodal request and let vLLM/the model processor apply Gemma's chat template:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://example.org/image.jpg"}},
{"type": "text", "text": "Describe this image accurately and concisely."},
],
}],
max_tokens=256,
)
print(response.choices[0].message.content)
Do not hand-insert raw image-placeholder text; the Gemma 4 processor/template is responsible for the image placement.
Public-safe provenance, calibration, artifact-audit, benchmark, and perplexity files are included in this repository:
provenance/calibration-and-source.jsonprovenance/artifact-audit.jsonbenchmarks/rtx3090-performance.jsonbenchmarks/perplexity.jsoneval/evaluate_gemma4_ppl_vllm.pyPlease cite the QAT/Heretic parent model card, the original Gemma 4 model card, and technical report for their respective methods and architecture. Google DeepMind developed Gemma; useful-quants packaged this W4A16 artifact from the QAT/Heretic parent checkpoint.
This repository inherits the parent Apache-2.0 license; see LICENSE and the Gemma 4 license.