Downloads · 30 days
92
1% of all-time downloads
sakamakismile/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4
Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 is a image-text-to-text model from sakamakismile. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as gemma.
NVFP4 quantized version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 — an abliterated (uncensored) Gemma 4 31B dense model with multimodal capability.
Downloads · 30 days
92
1% of all-time downloads
All-time downloads
11.7K
Public
Parameters
18.5B
20.5 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors20.4 GB · 100%
How the weights are stored.
U814.6B · 79%
From the Hugging Face model README
NVFP4 quantized version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 — an abliterated (uncensored) Gemma 4 31B dense model with multimodal capability.
59 GB → 20.5 GB. Single NVIDIA Blackwell GPU.
Gemma 4 31B Dense uses all 31B parameters on every token — unlike the 26B-A4B MoE variant (3.8B active). This means:
| Base model | huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 |
| Architecture | Gemma 4 Dense — 31B parameters, 60 layers |
| Quantization | NVFP4 W4A4 (weights FP4, activations FP4, scales FP8) |
| Format | compressed-tensors (native vLLM support) |
| Tool | vllm-project/llm-compressor (main) |
| Calibration | 32 samples, neuralmagic/calibration, seq_len=2048 |
| Size | 20.5 GB |
| Max context | 262,144 tokens |
| Requires | NVIDIA Blackwell GPU (SM 120) |
vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
--max-model-len 32768 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice
docker run --gpus '"device=0"' -p 8016:8016 \
-v /path/to/model:/models/current:ro \
--shm-size 16gb \
vllm/vllm-openai:cu130-nightly \
vllm serve /models/current --port 8016 --max-model-len 32768
Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).
| Test | Speed | Tokens | Result |
|---|---|---|---|
| English (CAP theorem) | 51.4 tok/s | 256 | PASS |
| Code (async fetcher) | 51.4 tok/s | 512 | PASS |
| Math (Bayes' theorem) | 51.3 tok/s | 512 | PASS |
| Container burst (×3) | 51.6 tok/s | 256 | PASS — stable |
Sustained: ~51 tok/s (single GPU, CUDA Graph PIECEWISE).
Slower than Gemma 4 26B-A4B MoE (~160 tok/s) because all 31B parameters are active per token vs. 3.8B for MoE. Trade-off: deeper reasoning per token.
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
targets="Linear",
scheme="NVFP4",
ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)
Following RedHatAI/gemma-4-31B-it-NVFP4 proven recipe.
from transformers import Gemma4ForConditionalGeneration, AutoProcessor
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch
MODEL_ID = "huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2"
model = Gemma4ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
recipe = QuantizationModifier(
targets="Linear", scheme="NVFP4",
ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)
ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:32]")
def preprocess(example):
messages = [{"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
for m in example["messages"]]
return processor.apply_chat_template(messages, return_tensors="pt", padding=False,
truncation=True, max_length=2048, tokenize=True, add_special_tokens=False,
return_dict=True, add_generation_prompt=False)
ds = ds.map(preprocess, batched=False, remove_columns=ds.column_names)
def data_collator(batch):
assert len(batch) == 1
return {k: (torch.tensor(v) if k != "pixel_values"
else torch.tensor(v, dtype=torch.bfloat16).squeeze(0))
for k, v in batch[0].items()}
oneshot(model=model, recipe=recipe, dataset=ds,
max_seq_length=2048, num_calibration_samples=32, data_collator=data_collator)
model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")
| Package | Version |
|---|---|
| torch | 2.11.0+cu130 |
| transformers | 5.5.4 |
| llmcompressor | 0.1.dev (main @ 3084520) |
| compressed-tensors | 0.15.1a20260414 |
| CUDA | 13.0 |
bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge