Downloads · 30 days
85
15% of all-time downloads
Gemma-4-E4B-it-MXFP4-GGUF
FreedomAISVR/Gemma-4-E4B-it-MXFP4-GGUF
Gemma-4-E4B-it-MXFP4-GGUF is a machine learning model from FreedomAISVR. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for llama.cpp. The card lists the license as apache-2.0.
Quantized GGUF release of google/gemma-4-E4B-it by FreedomAISVR. Ready for local inference with llama.cpp.
All-time downloads
562
Public
Repo size
6.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf6.1 GB · 100%
At a glance
- Library
- llama.cpp
- License
- apache-2.0
- Access
- Public
- Created
- Sep 18, 2026
- Updated
- Sep 18, 2026
- SHA
- 1b04391f
Base models
Model card
From the Hugging Face model README
Gemma 4 E4B IT — MXFP4 GGUF
Quantized GGUF release of google/gemma-4-E4B-it by FreedomAISVR. Ready for local inference with llama.cpp.
About the Base Model
| Property | Value |
|---|---|
| Model | google/gemma-4-E4B-it |
| Architecture | Gemma4ForConditionalGeneration (Vision + Text) |
| Parameters | ~3B (E4B = Efficient 4B-class) |
| Context Length | 131,072 tokens (128K) |
| Vision Support | Yes — image and video input via mmproj |
| License | Apache 2.0 |
| Developer | Google DeepMind |
Quantization Details
| Property | Value |
|---|---|
| Format | MXFP4 (OCP Microscaled FP4) |
| Bits Per Weight | 5.37 BPW |
| GGUF File Size | 4.8 GB |
| F16 Source | 15.0 GB (720 tensors) |
Files
| File | Size | Description |
|---|---|---|
gemma-4-e4b-it-mxfp4.gguf | 4.8 GB | MXFP4 quantized text model |
mmproj-gemma-4-E4B-it-f16.gguf | 945 MB | F16 multimodal projector (vision encoder) |
Performance (RTX 5060 Ti 16GB, CUDA 13.2)
| Metric | Value |
|---|---|
| Generation Speed | 97.1 tokens/sec |
| Prompt Processing | 175.1 tokens/sec |
| Context Length | 128K (Q8_0 KV cache) |
| VRAM Usage | ~7 GB total (model + KV + vision) |
Recommended llama.cpp Setup
These are the exact flags used to achieve the benchmarked speeds. Tested on RTX 5060 Ti 16GB with CUDA 13.2 and llama.cpp latest master.
Build llama.cpp (Windows)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build --config Release -j
Run with Full GPU Offload + Flash Attention + 128K Context
llama-cli -m gemma-4-e4b-it-mxfp4.gguf \
-ngl 99 \
--flash-attn on \
-c 131072 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--parallel 1 \
--jinja
Flag Explanation
| Flag | Purpose |
|---|---|
-ngl 99 | Offload all layers to GPU |
--flash-attn on | Enable Flash Attention (required for long context) |
-c 131072 | Set context to 128K tokens |
--cache-type-k q8_0 | Quantize KV key cache to Q8_0 (saves VRAM, near-lossless) |
--cache-type-v q8_0 | Quantize KV value cache to Q8_0 |
--parallel 1 | Single sequence (saves VRAM) |
--jinja | Enable Jinja2 chat template rendering |
Using the Vision Model (mmproj)
llama-server -m gemma-4-e4b-it-mxfp4.gguf \
--mmproj mmproj-gemma-4-E4B-it-f16.gguf \
-ngl 99 \
--flash-attn on \
-c 131072 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--port 8080
Then visit http://localhost:8080 for the web UI with image upload support.
Using with llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="gemma-4-e4b-it-mxfp4.gguf",
n_gpu_layers=99,
flash_attn=True,
n_ctx=131072,
verbose=False,
)
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}]
)
print(output["choices"][0]["message"]["content"])
Hardware Requirements
- GPU: NVIDIA RTX 50 series (Blackwell) recommended for MXFP4 acceleration
- VRAM: 8GB+ recommended (128K context with Q8_0 KV)
- RAM: 16GB+ system RAM
- Storage: 4.8 GB free space
License
This model is a quantized release of google/gemma-4-E4B-it, which is licensed under Apache 2.0. This quantized version inherits the same license.
Links
- Base Model: google/gemma-4-E4B-it
- FreedomAISVR: HuggingFace
- llama.cpp: GitHub
- Library
- llama.cpp
- License
- apache-2.0
- Created
- Sep 18, 2026
- Updated
- Sep 18, 2026