gemma-4-31B-it-Q4_K_M-dequant-bf16-vllm

Gemma 4 31B IT Q4_K_M GGUF Dequantized BF16 for vLLM

This private artifact was converted from unsloth/gemma-4-31B-it-GGUF file gemma-4-31B-it-Q4_K_M.gguf into standalone Gemma4 text safetensors for Transformers and vLLM.

The converter streams GGUF tensors, dequantizes them to fp32, maps Gemma4 text tensor names, then casts once to BF16 before writing sharded safetensors. The output config is rewritten from the multimodal gemma4 wrapper to standalone gemma4_text / Gemma4ForCausalLM.

Validation

• GGUF dry-run mapping: 833 input tensors; 832 mapped, rope_freqs.weight skipped as runtime metadata.
• Output index: 832 HF keys, 12 safetensor shards, 61,394,690,680 bytes.
• Transformers load: passed as Gemma4ForCausalLM with tied embeddings and 30,697,345,280 BF16 parameters.
• Numeric spot-checks: passed with max_abs_vs_bf16=0 for sampled embeddings, final norm, layer norms, attention, and FFN tensors.
• Transformers chat-template generation smoke: passed.
• vLLM chat-completions smoke: passed on vLLM 0.23.0.

vLLM command used

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/model \
  --served-model-name gemma-4-31b-it-gguf-q4_k_m-bf16 \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.75 \
  --trust-remote-code

This is an instruction model; use the chat template or /v1/chat/completions for behavioral smoke tests.