Downloads · 30 days
97
9% of all-time downloads
valoomba/Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8
Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8 is a image-text-to-text model from valoomba. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
W8A8 INT8 quantized version of llmfan46/gemma-4-31B-it-uncensored-heretic using AMD Quark.
Downloads · 30 days
97
9% of all-time downloads
All-time downloads
1.1K
Public
Parameters
31.3B
33.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors33.3 GB · 100%
How the weights are stored.
I829.3B · 94%
From the Hugging Face model README
W8A8 INT8 quantized version of llmfan46/gemma-4-31B-it-uncensored-heretic using AMD Quark.
| Field | Value |
|---|---|
| Base Model | llmfan46/gemma-4-31B-it-uncensored-heretic |
| Architecture | Gemma4ForConditionalGeneration (multimodal: text + vision) |
| Parameters | 31 B text decoder (quantized) + vision tower & embeddings kept in BF16 |
| Quantization | W8A8 INT8 (per-channel weight + per-token dynamic activation) |
| Quantizer | AMD Quark 0.11.2 (ptpc_int8 scheme, pack_method='order') |
| Model Size | ~33.3 GB (2 safetensors shards) |
| Original Size | ~62.5 GB (BF16 source checkpoint) |
| Compression | ~1.9x size reduction |
| Component | dtype | Granularity | Mode |
|---|---|---|---|
| Weight | INT8 | per-channel (ch_axis=0) | symmetric, static |
| Activation | INT8 | per-token (ch_axis=1) | symmetric, dynamic |
lm_head | BF16 | — | unquantized |
embed_tokens | BF16 | — | unquantized |
vision_tower / embed_vision | BF16 | — | unquantized (multimodal preserved) |
GSM8K 8-shot evaluation on the full 1319-question test split using vLLM's
OpenAI-compatible chat API (temperature=0, concurrency=16, max_tokens=512,
standard chat template, and final-answer format #### <answer>). Values are
only populated when this card is regenerated with EVAL_BASELINE_JSON and/or
EVAL_QUANT_JSON from eval_gsm8k_chat_vllm.py.
| Model | Scheme | Accuracy | Correct |
|---|---|---|---|
llmfan46/gemma-4-31B-it-uncensored-heretic (BF16 baseline) | — | not measured by this script | — |
| This model (Quark W8A8 INT8) | per-channel weight + per-token act. | not measured by this script | — |
# Generic vLLM serve command. Use TP=4 for 4x24GB RDNA3 cards.
vllm serve valoomba/Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8 \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9 \
--limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' \
--trust-remote-code
# On the local RDNA3/gfx1100 vLLM fork, use the optimized INT8 Triton path:
vllm serve valoomba/Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8 \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9 \
--limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' \
--trust-remote-code \
--linear-backend triton
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "valoomba/Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8",
"messages": [{"role": "user", "content": "Hello! What is the capital of France?"}],
"max_tokens": 256,
"temperature": 0.7
}'
--tensor-parallel-size 4, and keep multimodal limits disabled for text-only
serving as shown above.--max-model-len / batch settings.This model was quantized using AMD Quark's per-token per-channel INT8 scheme:
lm_head, token embeddings, the full vision_tower, and
embed_vision remain BF16.pack_method='order', weight_format='real_quantized', Quark
metadata (quant_method='quark') → real INT8 weights with BF16/FP32 scales
depending on loader/export path; no fake-quant weights are stored.pip install amd-quark==0.11.1 huggingface_hub safetensors
HIP_VISIBLE_DEVICES="" ROCR_VISIBLE_DEVICES="" CUDA_VISIBLE_DEVICES="" \
MODEL_IN=llmfan46/gemma-4-31B-it-uncensored-heretic \
MODEL_OUT=./Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8 \
HF_REPO_ID=valoomba/Gemma-4-31B-it-uncensored-heretic-Quark-W8A8-INT8 \
python quantize_gemma4_heretic_quark_w8a8_cpu.py
The script builds the Quark config explicitly:
weight: INT8, per_channel, ch_axis=0, symmetric=True, is_dynamic=False
input: INT8, per_channel, ch_axis=1, symmetric=True, is_dynamic=True
and then runs:
quantizer.direct_quantize_checkpoint(
pretrained_model_path=MODEL_IN,
save_path=MODEL_OUT,
device="cpu",
)
If you use this model, please cite the original Gemma 4 release and the upstream source model:
@misc{google2026gemma4,
title = {Gemma 4},
author = {Google DeepMind},
year = {2026},
url = {https://huggingface.co/google/gemma-4-31B-it}
}
Upstream model: llmfan46/gemma-4-31B-it-uncensored-heretic.
This quantized derivative follows the license metadata of the upstream model,
which is listed as Apache 2.0 / Gemma 4 license metadata on Hugging Face. Check
the upstream model card and any included LICENSE / NOTICE files for complete
terms.
Modified files include the INT8-quantized safetensors checkpoint and the appended
Quark quantization_config block in config.json. No warranty of any kind is
provided.