Downloads · 30 days
32
23% of all-time downloads
Gemma-4-26B-A4B-text-base-fp8
wyattearp/Gemma-4-26B-A4B-text-base-fp8
Gemma-4-26B-A4B-text-base-fp8 is a text generation model from wyattearp. Use it when you need the model to write or continue text. The card lists the license as gemma.
Gemma-4-26B-A4B-text-base-fp8 is a post-training quantized (PTQ) FP8 release of Google's Gemma-4-26B-A4B-it Mixture-of-Experts (MoE) model. It is modality-stripped to a pure language model (Gemma4ForCausalLM) and quan…
All-time downloads
137
Public
Parameters
25.2B
26 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors26 GB · 100%
Parameter types
How the weights are stored.
F8_E4M324.5B · 97%
Try a prompt
Base models
Model card
From the Hugging Face model README
Google Gemma-4-26B-A4B-text-base-fp8 (ModelOpt Calibrated FP8 for Ada & L40S)
Gemma-4-26B-A4B-text-base-fp8 is a post-training quantized (PTQ) FP8 release of Google's Gemma-4-26B-A4B-it Mixture-of-Experts (MoE) model. It is modality-stripped to a pure language model (Gemma4ForCausalLM) and quantized using NVIDIA ModelOpt with per-tensor scalar scaling factors (axis=None) specifically engineered for native single-GPU serving on NVIDIA L40S (48 GB GDDR6), Ada Lovelace (RTX 4090 / SM 8.9), and Blackwell architectures.
⚡ Performance & Footprint
- Checkpoint Footprint: 24.2 GiB (down from 50.47 GiB in BF16, a 52% reduction).
- Hardware Target: NVIDIA L40S (48 GB), RTX 4090, RTX 6000 Ada, and Blackwell GB10 / B200.
- Unified Memory Headroom: Takes 24.6 GiB VRAM loaded in vLLM, leaving ~23.4 GiB free for KV cache on a single 48 GB L40S GPU.
- Architecture: Modality-stripped pure text backbone (
Gemma4ForCausalLM), eliminating multimodal projector and vision tower VRAM overhead. - Scales Format: 23,450 per-tensor scalar scaling factors, matching native Ada Lovelace FP8 Tensor Core hardware GEMM instructions (
torch._scaled_mm).
🔬 Quantization Topology
| Component | Target Precision | Technical Rationale |
|---|---|---|
| MoE Expert MLPs (128 Experts) | FP8 (E4M3, Per-Tensor Scales) | Calibrated across 64 diverse domain prompts. Per-tensor scalar scaling factors match hardware Tensor Core instructions. |
| Attention Layers ($q, k, v, o$) | FP8 (E4M3, Per-Tensor Scales) | Calibrated activation and weight scaling for high-throughput inference. |
| MoE Gating & Routers | Native BF16 | Preserves high dynamic range router logits to prevent expert routing collapse. |
| Embeddings & LM Head | Native BF16 | Preserves token entropy across the 256,000 vocabulary tokens. |
⚠️ Important: Tool-Calling Parser Bugfix in vLLM
When deploying Gemma-4 with tool calling in stock vLLM, tool calls can be dropped or fragmented when transitioning directly out of the thinking channel (e.g., <channel|>call:... without leading whitespace).
- Upstream Bug Report: vllm-project/vllm#54256
- Official Pull Request: vllm-project/vllm#54257 —
[Bugfix][Parser] Support bare call: and whitespace-free channel transitions in Gemma4 parser
🚀 Serving with vLLM
This model is in standard modelopt FP8 format and runs natively on single-GPU L40S or Blackwell setups:
vllm serve wyattearp/Gemma-4-26B-A4B-text-base-fp8 \
--served-model-name google/gemma-4-26B-A4B-text-base-fp8 \
--host 0.0.0.0 \
--port 8000 \
--dtype auto \
--trust-remote-code \
--gpu-memory-utilization 0.85 \
--max-model-len 262144 \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--enable-auto-tool-choice
📄 License
Inherits upstream Gemma Terms of Use from Google DeepMind: Gemma Terms of Use.
- Task
- Text Generation
- Type
- gemma4
- License
- gemma
- Languages
- en
- Created
- Sep 1, 2026
- Updated
- Sep 19, 2026