gemma-4-26B-A4B-heretic-NVFP4

Gemma-4-26B-A4B heretic NVFP4

NVFP4 quantization of llmfan46/gemma-4-26B-A4B-it-ultra-uncensored-heretic, produced for low-latency RP/storyline serving on NVIDIA Blackwell GPUs.

Model facts

Quantized layers: all Linear. Ignored: lm_head, re:.*embed.*, re:.*router, re:.*vision_tower.*.

Inference (vLLM)

vLLM auto-detects compressed-tensors NVFP4; no --quantization flag needed. Tested on vLLM 0.20.0 + CUDA 13.0 + RTX PRO 6000 Blackwell (96 GB).

vllm serve louismuk/gemma-4-26B-A4B-heretic-NVFP4 \
  --served-model-name gemma-4-26b \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --max-num-seqs 64 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 \
  --enable-prompt-tokens-details \
  --host 0.0.0.0 --port 8000

Performance

Multi-turn RP benchmark (sys=2048 / user=256 / out=256 tok, 5 turns/conversation, prefix-cache enabled). 1× RTX PRO 6000 Blackwell Server.

Cache-hit rate ~75 % across all scenarios (prefix cache works correctly with the MoE).

Quantization recipe

Produced with llm-compressor (git main) using its gemma4_example.py recipe — which uses SequentialGemma4TextExperts to walk Gemma-4's fused expert tensors (nvidia-modelopt 0.43's public configs do not handle these correctly).

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoProcessor, Gemma4ForConditionalGeneration

model = Gemma4ForConditionalGeneration.from_pretrained(
    "llmfan46/gemma-4-26B-A4B-it-ultra-uncensored-heretic", dtype="auto",
)
processor = AutoProcessor.from_pretrained("llmfan46/gemma-4-26B-A4B-it-ultra-uncensored-heretic")

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)
# calibration: neuralmagic/calibration LLM split, 20 samples, max_seq_length=8192
oneshot(model=model, recipe=recipe, dataset=ds,
        max_seq_length=8192, num_calibration_samples=20, data_collator=collator)
model.save_pretrained("gemma-4-26B-A4B-heretic-NVFP4", save_compressed=True)

Wall time: ~7 min on a single RTX PRO 6000 Blackwell.

Heretic / abliteration provenance

Inherits the abliteration applied by llmfan46/gemma-4-26B-A4B-it-ultra-uncensored-heretic (Heretic v1.2.0, Arbitrary-Rank Ablation). Refusal behaviours are substantially reduced. Use responsibly.

License

Apache 2.0 (inherited from base).