gemma-4-31B-it-fp8

gemma-4-31B-it-fp8

FP8 (W8A8 dynamic) quantization of google/gemma-4-31B-it, produced with llm-compressor. lm_head, the vision/audio towers, mlp.down_proj, self_attn.o_proj, and the first/last transformer layers are kept in the original precision - the layers most consistently found to be quantization-sensitive - to recover agentic/tool-calling quality.

Quality

KL divergence of the quantized model's next-token distribution vs the bf16 baseline (seq len 2048). prose is WikiText-2; agentic is tool-calling turns (schema + query + call) rendered with the chat template.

Like the smaller gemma-4-E2B-it-fp8, this holds up better on agentic/tool-calling text than on general prose - the opposite pattern from the Qwen quants. Worth weighing against your actual workload before deploying.

Usage

vLLM:

vllm serve mahadev9/gemma-4-31B-it-fp8

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mahadev9/gemma-4-31B-it-fp8", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/gemma-4-31B-it-fp8")