gemma4-E4B-opt

Gemma‑4 E4B — W4A16 GPTQ + 10% MLP pruning + FP8 KV cache

Team: WattsPruners
Kilian Pfister¹, Mauricio Fadel Argerich² ³, Jonathan Fürst¹ ³ <br> [email protected] <br> [email protected] / [email protected] <br> [email protected] / [email protected] <br> ¹ ZHAW (Zurich University of Applied Sciences), https://www.zhaw.ch <br> ² UPM (Universidad Politécnica de Madrid), https://www.upm.es/ <br> ³ Leaner AI – cost and energy-efficient AI https://leaner-ai.com/ <br>

Energy‑efficient, multimodal (image‑to‑text) build of google/gemma-4-E4B-it, compressed for single‑GPU (1×L4) serving under vLLM.

• Base model: google/gemma-4-E4B-it
• Recipe: w4a16_gptq_prune10_mlponly_kv_fp8e4m3
• Format: compressed-tensors / pack-quantized
• License: same as the base model (Gemma Terms of Use).

---

Compression techniques

This checkpoint stacks three independent compression steps. All of them are applied only to the language model — the vision tower, audio tower, multimodal projectors, embeddings and lm_head are left at full precision (see ignore list in config.json), because vLLM cannot serve a quantized vision/audio tower.

Note on the pruning step. The 10% sparsity is unstructured. vLLM ≥0.19 has no sparse kernel, so the weights are served dense — the pruning does not itself accelerate inference. Its role here is regularization/weight clean‑up ahead of GPTQ. The energy and footprint wins below come from the 4‑bit weights + FP8 KV cache, not from the sparsity.

The exact, machine‑readable recipe is in recipe.yaml / recipe.json and the quantization metadata in quantization.yaml.

---

Results vs. BF16 baseline

Both the baseline and this model were evaluated on a single NVIDIA L4, vLLM 0.19.0, with the identical serving profile (max_num_seqs=1024, FP8 E4M3 KV cache, prefix caching and chunked‑prefill on, speculative decoding off) and the same client concurrency (256), using greedy decoding (temperature=0). Energy was measured by NVML at 10 Hz and integrated over the run window with the trapezoidal rule, cross‑checked with CodeCarbon. Each configuration was run three times; values are mean ± 1 std. Because decoding is greedy, that spread is vLLM's system nondeterminism, not sampling noise.

Energy is the total GPU energy for one complete benchmark run. ΔE and ΔAcc. are relative to BF16 and computed from unrounded means.

The energy roughly halves on both MMMU benchmarks while accuracy drops by at most 2.9 pp; on MedXpertQA accuracy is unchanged within the noise. Across the four suites the model uses 1.7–2.8× less GPU energy per run (2.07× on MMMU‑Pro, the primary optimisation benchmark).

Suite sizes: MMMU 900 (validation, all 30 subjects) · MMMU‑Pro 3,460 (standard 10‑option + vision) · MATH‑Vision 304 (testmini) · MedXpertQA 470 of 500 (30 samples exceed the 3‑image serving cap and are excluded identically for both models). Generation caps: 1024 tokens, except MATH‑Vision at 2048.

Footprint

¹ Peak VRAM is essentially unchanged because vLLM reserves ~90% of the GPU regardless; the real memory win is the 37.5% smaller checkpoint and the FP8 KV cache freeing space for KV blocks.

How to read the energy number

Average board power is essentially the same for both models (≈67–72 W) — the GPU does not draw less power, it finishes sooner. The identical workload completes 1.6–2.8× faster, so proportionally fewer GPU‑seconds (and joules) are spent to produce the same answers.

The saving is therefore entirely a per‑token efficiency win: energy per token drops by 42–65% depending on the suite (−53% on MMMU‑Pro), which tracks the per‑run saving almost exactly. On a synthetic fixed‑length throughput benchmark that isolates this term, W4A16 GPTQ alone cuts energy per generated token by 42.7% (67.2 → 38.5 mJ/token, throughput 398 → 718 tok/s). Serving‑side tuning adds to it, mostly via the FP8 KV cache (≈−15%): it halves the bytes streamed from HBM per decode step, and since decoding is memory‑bandwidth‑bound, that shortens the run at unchanged power.

The trade is at most 2.9 accuracy points, which is mostly the cost of 4‑bit weights; the 10% MLP pruning and the FP8 KV cache add little degradation on top. All four suites stay well above the Resilient AI Challenge's 80%‑of‑baseline accuracy threshold.

---

Serving

The evaluator launches the model as:

vllm serve <model-uri> --config vllm_config.yaml

The bundled vllm_config.yaml pins the tuned profile:

gpu-memory-utilization: 0.9
max-model-len: 20000
max-num-seqs: 1024
max-num-batched-tokens: 4096
enable-prefix-caching: true
enable-chunked-prefill: true
quantization: compressed-tensors
kv-cache-dtype: fp8_e4m3
limit-mm-per-prompt:
  image: 3

Sampling settings used for all reported evaluations: greedy decoding (temperature=0, top_p=1.0). The bundled generation_config.json keeps the base model's own defaults (temperature=1.0, top_p=0.95, top_k=64) for general use.

---

Provenance

• Hardware: 1×NVIDIA L4 (sm89), CUDA 12.8
• Stack: vLLM 0.19.0 · transformers 5.5.4 · llmcompressor 0.10.1.dev148 · torch 2.10.0+cu128
• Quantization cost: ~4,840 s wall‑clock, 16.8 GB peak VRAM (one‑shot, offloaded Hessians)
• Calibration data: ultrachat_200k (train_sft), 128 samples, max seq len 256