gemma-4-E2B-it-fp8

gemma-4-E2B-it-fp8

FP8 (W8A8 dynamic) quantization of google/gemma-4-E2B-it, produced with llm-compressor. lm_head, the vision/audio towers, mlp.down_proj, self_attn.o_proj, and the first/last transformer layers are kept in the original precision - the layers most consistently found to be quantization-sensitive - to recover agentic/tool-calling quality.

Quality

KL divergence of the quantized model's next-token distribution vs the bf16 baseline (seq len 2048). prose is WikiText-2; agentic is tool-calling turns (schema + query + call) rendered with the chat template.

Holds up much better than gemma-4-E2B-it-nvfp4 (overall mean KL 0.73, 76.1% top-1) - recommended over the 4-bit version for this model. Like the other Gemma-4 quants, it does better on agentic/ tool-calling text than on general prose, the opposite pattern from the Qwen quants.

Usage

vLLM:

vllm serve mahadev9/gemma-4-E2B-it-fp8

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mahadev9/gemma-4-E2B-it-fp8", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/gemma-4-E2B-it-fp8")