gemma-4-E2B-it-nvfp4

gemma-4-E2B-it-nvfp4

NVFP4 (W4A4) quantization of google/gemma-4-E2B-it, produced with llm-compressor. lm_head, the vision/audio towers, mlp.down_proj, self_attn.o_proj, and the first/last transformer layers are kept in the original precision - the layers most consistently found to be quantization-sensitive - to recover agentic/tool-calling quality.

Quality

KL divergence of the quantized model's next-token distribution vs the bf16 baseline (seq len 2048). prose is WikiText-2; agentic is tool-calling turns (schema + query + call) rendered with the chat template.

Significant degradation remains at 4-bit even with the sensitive-layer exclusions above, especially on general prose - this is a small (E2B) model and 4-bit weights+activations hit it hard. For most uses prefer gemma-4-E2B-it-fp8, which is far closer to the baseline.

Usage

vLLM:

vllm serve mahadev9/gemma-4-E2B-it-nvfp4

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mahadev9/gemma-4-E2B-it-nvfp4", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/gemma-4-E2B-it-nvfp4")