Downloads · 30 days
62
27% of all-time downloads
Zek-Takai/gemma-4-26B-A4B-it-NVFP4-fp8global
gemma-4-26B-A4B-it-NVFP4-fp8global is a text generation model from Zek-Takai. Use it when you need the model to write or continue text. The card lists the license as gemma.
NVFP4 build with the output head quantized instead of the usual BF16 (ignore=["lmhead"]). Loads and serves in stock vLLM — no source patches — where ModelOpt-format quantized heads currently fail (vllm-project/vllm 44…
Downloads · 30 days
62
27% of all-time downloads
All-time downloads
232
Public
Parameters
14.4B
17 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.9 GB · 100%
How the weights are stored.
U812.9B · 89%
From the Hugging Face model README
NVFP4 build with the output head quantized instead of the usual BF16
(ignore=["lm_head"]). Loads and serves in stock vLLM — no source patches —
where ModelOpt-format quantized heads currently fail
(vllm-project/vllm #44081, #45102). Per-token weight reads: 3.23 GB → 2.17 GB.
Full recipe in recipe/, raw benchmark tables in
data/.
All measurements: DGX Spark (GB10, ~273 GB/s), vLLM 0.26-series dev, CUDA graphs on, marlin MoE, fp8 KV. Full flags below.
Multi-hour session through an agent framework (~18k-token system prompt, persona loaded, tools on, prefix caching). ~28 requests, 15.7k tokens generated, context to 59k. Decode from engine ITL counters:
| Resident ctx | Generation | TTFT | tok/s |
|---|---|---|---|
| 22k cold | 899 | 5.6 s | 67.0 |
| 22k warm | 517 | 0.13 s | 67.2 |
| 50k (15k fresh) | 709 | 11.3 s | 61.4 |
| 51k warm | 2,052 | 0.14 s | 61.4 |
| 59k | 4,553 (one answer) | 0.85 s | 59.7 |
Scripted 11-turn document session (two ~20k-token docs, code gen, cross-doc analysis): ~64 tok/s prefill-corrected at 45k, single 5,529-token answer.
Single-stream, accumulating conversation, no output caps:
| Resident ctx | 26 tok | 2.8k | 6.3k | 13k | 26k | 40k | 53k |
|---|---|---|---|---|---|---|---|
| tok/s | 80.5 | 77.4 | 75.8 | 72.5 | 67.7 | 64.3 | 62.7 |
Same rig, same flags, only the head differs. Comparison arm: a publicly available NVFP4 build of the same base with the standard BF16 head. TPOT-derived decode; fixed output length so the arms are strictly comparable:
| Input | 2k | 4k | 8k | 16k | 32k | 65k |
|---|---|---|---|---|---|---|
| quantized head | 74.2 | 73.4 | 71.2 | 68.9 | 63.8 | 59.5 |
| BF16 head | 52.1 | 51.7 | 50.6 | 49.3 | 46.7 | 44.1 |
+35–42% at every depth; the ladder and the live session agree at matching
depths. A third arm (uniform-NVFP4 control with the same quantized head) is
in data/decode_ladder_head_ab.csv.
enable_thinking off/on, same prompts, natural stops: 79.7 → 77.9 tok/s and
78.0 → 77.1 tok/s. Thinking costs volume (1.8–2.5× tokens), not rate.
data/mtp_context_ladder.csv.Published in full: recipe.yaml as recorded by the build, and
recipe/quant_fp8global.py — the complete
script. Scheme: NVFP4 on every Linear including lm_head; FP8_DYNAMIC on
the attention projections of the five global-attention layers (5, 11, 17, 23,
29), where massive-activation outliers would blow out NVFP4 block scales; BF16
router/embeddings/norms. Three post-quant config/tokenizer fixups make the
quantized head loadable in stock vLLM — recipe/README.md
documents each.
Machine-readable copies of every table above, plus recorded results without a
table, in data/:
decode_ladder_head_ab.csv — the head A/B
ladder, with a third arm the table above omits: a uniform-NVFP4 control
carrying the same quantized head. It isolates the head from the
FP8-global-layers choice — the uniform control runs ~4–5% faster than this
build (fewer FP8 weight bytes), both arms hold +35–42% over the BF16 head.context_ladder_unbounded.csv — the
unbounded single-stream context ladder.live_agent_session.csv — the live agent
session, per-request: resident context, generation length, TTFT, decode rate.mtp_context_ladder.csv — MTP drafter off/on
at every context rung. Shows the ~2k crossover: +11% at tiny context,
progressively unprofitable beyond it (drafter re-attends the full KV per
draft step; acceptance stays healthy throughout — it is a compute cost, not
acceptance decay).data/README.md — methodology, dates, and the recorded
results that have no CSV: thinking on/off rates, MTP acceptance band,
tool-calling 8/8 with byte-identical arguments, quality probes vs the
BF16-head control.Requirements: a Blackwell-class GPU and vLLM 0.26+ — its compressed-tensors 0.17 reader is what loads this container (older engines cannot). Measured on a 0.26-series dev build, GB10/SM121.
hf download Zek-Takai/gemma-4-26B-A4B-it-NVFP4-fp8global --local-dir gemma4-nvfp4-qhead
Serve — stock vLLM, no source modifications required
(examples/serve_vllm.sh is this command as a
script):
vllm serve ./gemma4-nvfp4-qhead \
--served-model-name gemma4-nvfp4-quantized-head \
--moe-backend marlin \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--max-model-len 252000 \
--kv-cache-dtype fp8 \
--kv-cache-memory-bytes $((26 * 1024**3))
Flag notes: --moe-backend marlin is what was measured on SM121;
--enable-auto-tool-choice --tool-call-parser gemma4 enables tool calling;
the absolute KV budget is unified-memory hygiene (GPU-fraction settings
overcommit shared pools) — discrete-VRAM GPUs can use standard settings.
Query — OpenAI-compatible endpoint; thinking is controlled per request via the chat template:
curl -s http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma4-nvfp4-quantized-head",
"messages": [{"role": "user", "content": "Hello"}],
"chat_template_kwargs": {"enable_thinking": false}
}'
patches/)Base serving needs no vLLM changes. Two optional lanes do — apply into your vLLM install, and re-apply after any vLLM upgrade:
VLLM_ROOT=$(python -c "import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent.parent)")
Completions-style pre-templated input —
patches/vllm_bos_dedupe/ collapses the
doubled leading BOS such input produces, which measurably degrades gemma-4
(chat requests are unaffected; skip this if you only use
/v1/chat/completions):
patch -p1 -d "$VLLM_ROOT" < patches/vllm_bos_dedupe/bos_dedupe.patch
Speculative decoding with Google's official MTP drafter —
patches/vllm_gemma4_mtp_embed_share/
lets vLLM share the target's embedding with the
gemma-4-26B-A4B-it-assistant drafter (upstream-reported; see patch
README). Then serve with:
patch -p1 -d "$VLLM_ROOT" < patches/vllm_gemma4_mtp_embed_share/embed_share.patch
--speculative-config '{"model": "google/gemma-4-26B-A4B-it-assistant", "method": "gemma4_mtp", "num_speculative_tokens": 1}'
Worth it only under ~2k resident context on bandwidth-bound hardware —
see data/mtp_context_ladder.csv.
Quantization recipe, benchmarks, and documentation co-authored with Claude (Anthropic). Recipe code and benchmark data: Apache-2.0.
Derived from google/gemma-4-26B-A4B-it; subject to the Gemma Terms of Use.