gemma-4-31b-it-nvfp4-hybrid

Gemma 4 31B IT — NVFP4 hybrid verifier (primary model)

This is the main model. Pair it with undeadindustries/gemma-4-31b-it-dflash-drafter-nvfp4 for optional speculative-decoding speedup on vLLM.

The pair — what to load together

Use together: load the verifier as your main vllm serve model and point --speculative-config at the drafter. The drafter was trained on hidden states from this exact quant — a generic drafter trained on BF16 Gemma will not work well (acceptance collapses).

Verifier-only is fine if you only want quality + smaller footprint (~30 GB vs ~62 GB BF16). You do not need the drafter for correct answers.

Why this quant exists

Hand-tuned hybrid NVFP4 of google/gemma-4-31b-it for instruction following and long-document work on memory-constrained Blackwell boxes (e.g. 128 GB unified memory):

• BF16 attention (q/k/v/o) — protects dynamic range for 256K-class reasoning and needle-in-haystack retrieval
• NVFP4 W4A4 MLP (gate/up/down) — ~2× weight compression on the bulk of parameters
• Text-only — vision stripped; Gemma4ForCausalLM architecture

Design goal: smallest practical footprint that keeps -it instruction behavior, not maximum compression at any quality cost.

Quantization scheme

Built with AutoRound scheme=NVFP4, exported as llm-compressor / nvfp4-pack-quantized for vLLM.

Verified quality (vLLM 0.20.2, GB10)

Greedy /v1/chat/completions (required for -it models):

• 8/8 prompt battery: factual, arithmetic, reasoning, code, instruction-following, knowledge, translation
• 13.7K-token needle-in-haystack: pass
• Native NVFP4 GEMM on Blackwell (FlashInferCutlassNvFp4LinearKernel)

Serve verifier only

vllm serve undeadindustries/gemma-4-31b-it-nvfp4-hybrid \
  --port 8000 --max-model-len 4096 \
  --gpu-memory-utilization 0.55 --enforce-eager --trust-remote-code \
  --generation-config vllm

Serve verifier + drafter together (recommended pair)

vllm serve undeadindustries/gemma-4-31b-it-nvfp4-hybrid \
  --port 8000 --max-model-len 4096 \
  --gpu-memory-utilization 0.55 --enforce-eager --trust-remote-code \
  --generation-config vllm \
  --max-num-batched-tokens 8192 --max-num-seqs 64 \
  --speculative-config '{"model":"undeadindustries/gemma-4-31b-it-dflash-drafter-nvfp4","method":"dflash","num_speculative_tokens":15}'

Requires vLLM ≥ 0.20.2 (e.g. vllm-node:latest). Combined weights: ~36 GB VRAM.

Observed speculative acceptance_length ~1.19 with the paired drafter — modest speedup, no quality regression when using chat API.

Memory note

~30 GB = weights. vLLM may reserve ~55–67 GB total for KV cache and batch slots at default utilization — that is headroom, not model size.

License

Derived from google/gemma-4-31b-it (Apache 2.0). Subject to Gemma Terms of Use.