Huihui-gemma-4-26B-A4B-it-abliterated-FP8-Dynamic

Huihui-gemma-4-26B-A4B-it-abliterated FP8-Dynamic

vLLM-compatible FP8-Dynamic quantization of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated.

The original BF16 release (~50 GB) is not directly servable by vLLM at production speed — --quantization fp8 runtime path is ~6× slower than pre-quantized FP8. This repo fills that gap: 27 GB FP8 weights, drop-in for vLLM, multimodal vision retained, abliteration profile preserved.

Smoke test (DGX Spark / GB10)

Speculative-decoding bench (DGX Spark, GB10, batch=1, T=0.0)

Tested pairing this model as MTP target with the vanilla google/gemma-4-26B-A4B-it-assistant draft model:

Verdict: MTP gives a real +33% gain even with vanilla draft head. num_speculative_tokens=1 is recommended — cleaner per-step acceptance (69% vs 40% token-level for n=4), same throughput. The per-position decay (65% → 21% over 4 positions) is consistent with huihui's abliteration shifting the body's prediction distribution; n=1 only uses the high-confidence first position.

For comparison, vanilla Gemma 4 + MTP achieves 108 tok/s on the same hardware (Part 27 reference). The ~50% gap is the "abliteration tax" — likely from a combination of FP8 calibration on shifted weights and MoE routing imbalance after abliteration. If you need raw speed without abliteration, use the vanilla google/gemma-4-26B-A4B-it directly.

vLLM serving

Recommended (with MTP, +34%)

vllm serve coolthor/Huihui-gemma-4-26B-A4B-it-abliterated-FP8-Dynamic \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":1}' \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.65 \
  --max-model-len 8192 \
  --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --trust-remote-code

Note: MTP requires vLLM with PR #41745's gemma4_mtp.py integration. Tested with the vllm/vllm-openai:gemma4-0505-arm64-cu130 image.

Usage (client side)

⚠️ This is an instruction-tuned model. Always use the chat completions endpoint.

curl -s -X POST http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "huihui-gemma4",
    "messages": [{"role": "user", "content": "Write a haiku about an old API."}],
    "max_tokens": 80,
    "temperature": 0.7
  }'

Do NOT use /v1/completions with a raw prompt — it bypasses the Gemma chat template (<start_of_turn>user ... <end_of_turn>\n<start_of_turn>model\n) and the model will fall off-distribution and loop garbage tokens. This is not abliteration / FP8 / MTP related; vanilla google/gemma-4-26B-A4B-it behaves the same way under /v1/completions. If you must use the completions endpoint, apply the chat template manually.

Quantization recipe

Used llmcompressor with FP8_DYNAMIC scheme. Critical ignore list — note re:.*router.* in particular: omitting this entry quantizes MoE router weights and breaks expert dispatch (verified by trial in our own testing):

ignore = [
    "re:.*router.*",        # MoE router — must NOT quantize
    "lm_head",
    "re:.*embed_tokens.*",
    "re:.*norm.*", "re:.*layernorm.*", "re:.*layer_norm.*",
    "re:.*rmsnorm.*", "re:.*rms_norm.*",
    "re:.*conv1d.*", "re:.*linear_attn.*",
    "re:visual.*", "re:model.visual.*",
    "re:.*patch_embed.*", "re:.*vision.*", "re:.*image.*",
    "re:.*video.*", "re:.*projector.*", "re:.*merger.*",
    "re:.*mlp.gate$", "re:.*shared_expert_gate.*",
    "re:.*embed_audio.*", "re:.*embed_vision.*",
    "re:.*audio_tower.*", "re:.*audio_projector.*",
]

Quantization run: 2.9 minutes on GB10 (data-free pipeline, 30 MoE module calibration, FP8 scale derivation).

Toolchain notes

llmcompressor stable releases pin transformers <= 4.57.6, but Gemma4ForConditionalGeneration requires transformers 5.0+. Workaround: patch llmcompressor/entrypoints/utils.py to remove deprecated use_auth_token=... kwargs (8 occurrences) before reinstalling editable.

Credits

• Base abliteration: huihui-ai
• Original Gemma 4: google/gemma-4-26B-A4B-it
• MTP draft model: google/gemma-4-26B-A4B-it-assistant
• Quantization toolchain: vllm-project/llm-compressor

License

Apache 2.0, inherited from base model and Gemma usage terms.

---

☕ If this saved you GPU hours, you can buy me a coffee.

---

📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.

---

Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.