gemma-4-26B-A4B-it-NVFP4

Gemma-4-26B-A4B-it-NVFP4

NVFP4 quantization of google/gemma-4-26b-a4b-it — the Mixture-of-Experts variant of Gemma 4 with 25.2B total parameters and ~3.8B active per token. Quantized using NVIDIA TensorRT Model Optimizer (modelopt) on DGX Spark hardware.

Supports text, image, and multimodal inference with a 256K token context window.

Base Model vs. NVFP4 Comparison

Quantization Details

Quality-Preserving Exclusions

The following layers are kept at full bfloat16 precision to preserve output quality:

This is standard best practice for MoE quantization: compress the bulk of the transformer (attention + FFN layers) where FP4 has minimal quality impact, but keep sensitive routing, vision, and output layers at full precision. The tradeoff is a slightly larger model (~16 GB vs theoretical ~12 GB for full FP4) but significantly better quality retention.

MoE Quantization Challenge

Gemma 4 MoE stores expert weights as fused 3D tensors (shape [128, dim, dim]) rather than individual linear modules. NVIDIA modelopt only quantizes linear layers — the fused expert parameters (which are ~91% of the model) require a custom plugin to unfuse into individual linear layers before quantization.

Hardware

Quantized and serving on NVIDIA DGX Spark:

• GB10 Grace Blackwell processor
• 128 GB unified CPU+GPU memory
• CUDA 13.1
• SM 12.1

Model loads at ~16 GB, leaving substantial headroom for long-context inference with KV cache.

Serving with vLLM

Quick Start

vllm serve CyberFitz/gemma-4-26B-A4B-it-NVFP4 \
  --quantization modelopt \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 \
  --moe-backend marlin \
  --trust-remote-code

Key Flags

Docker (DGX Spark)

docker run -d \
  --name gemma4-nvfp4 \
  --gpus all --ipc=host --network host \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  -v /path/to/model:/model \
  <your-vllm-image> \
  vllm serve /model \
    --quantization modelopt \
    --max-model-len 262144 \
    --kv-cache-dtype fp8 \
    --moe-backend marlin \
    --trust-remote-code

Testing

# Text generation
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-26B-A4B-it-NVFP4",
    "messages": [{"role": "user", "content": "Explain mixture of experts in one paragraph."}],
    "max_tokens": 200
  }'

Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "CyberFitz/gemma-4-26B-A4B-it-NVFP4",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("CyberFitz/gemma-4-26B-A4B-it-NVFP4")

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Benchmarks

Coming soon — GSM8K, IFEval, and throughput benchmarks comparing BF16 vs NVFP4 on DGX Spark.

License

This model inherits the Gemma license from the base model.

Credits

Quantized by CyberFitz on NVIDIA DGX Spark hardware.