gemma-4-31B-it-GGUF-SPLIT

Gemma 4 31B Dense — GGUF Split

This repository contains split GGUF shards of unsloth/gemma-4-31B-it-GGUF, optimized for use with llama.cpp (CPU & GPU inference). Files are split into ≤5 GB shards for easier downloading and HuggingFace LFS compatibility.

Inspired by Nielk38/Qwen3.6-35B-A3B-GGUF-SPLIT. See also Nielk38/gemma-4-26B-A4B-it-GGUF-SPLIT.

Available Quantizations

CPU benchmark: AMD Ryzen 7 9800X3D (8c/16t, AVX-512 Zen4 backend), 16 threads, CPU-only (no GPU offload). Dense 31B — all parameters active, ~5x slower than Gemma 4 26B A4B MoE (52.9/12.3 t/s).

Multimodal Projection Files

How to Download & Use

Download shards

huggingface-cli download Nielk38/gemma-4-31B-it-GGUF-SPLIT \
  --include "*.gguf" \
  --local-dir ./gemma4-31b-gguf

Run with llama.cpp (first shard auto-loads all)

llama-cli -m ./gemma4-31b-gguf/gemma-4-31B-it-Q4_K_M-00001-of-00004.gguf \
  --mmproj ./gemma4-31b-gguf/mmproj-BF16.gguf \
  -ngl 0 -t 16 -p "What is 2+2?" -n 128

Merge shards (optional)

llama-gguf-split --merge \
  ./gemma4-31b-gguf/gemma-4-31B-it-Q4_K_M-00001-of-00004.gguf \
  ./gemma-4-31B-it-Q4_K_M.gguf

---

Benchmark Comparison: Gemma 4 31B (Dense) vs Gemma 4 26B A4B (MoE) vs Qwen3.6 35B A3B (MoE)

Key distinction: Gemma 4 31B is a dense model — all 30.7B parameters active at inference, so it is slower but higher quality than the 26B A4B MoE which only activates 3.8B parameters.

Gemma 4 31B edges out the 26B A4B on all benchmarks due to full parameter utilization, but runs ~2-3x slower on CPU. Qwen3.6 35B A3B leads on pure reasoning/math tasks despite only 3B active params.

---

Original unsloth README

---

license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text base_model: google/gemma-4-31B-it tags:

• gemma4
• unsloth
• gemma
• google

---

Read our How to Run Gemma 4 Guide!

<div> <p style="margin: 0 0 0px 0; margin-top: 0px;"> <em>See <a href="https://unsloth.ai/docs/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0 GGUFs</a> for our quantization benchmarks.</em> </p> <div style="display: flex; gap: 5px; align-items: center; margin-bottom: 0px;"> <a href="https://github.com/unslothai/unsloth/"> <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133"> </a> <a href="https://discord.gg/unsloth"> <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173"> </a> <a href="https://unsloth.ai/docs/models/gemma-4"> <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143"> </a> </div> <ul style="margin: 0;"> <li><b>Apr 11 Update:</b> Re-download for Google's latest chat template and llama.cpp fixes.</li> <li>Gemma 4 can now be run and fine-tuned in <a href="https://unsloth.ai/docs/new/studio">Unsloth Studio</a>. <a href="https://unsloth.ai/docs/models/gemma-4">Read our guide</a>.</li> <li>See all versions of Gemma 4 (GGUF, 16-bit etc.) <a href="https://huggingface.co/collections/unsloth/gemma-4">in our collection</a>.</li> </ul> </div>

---

<div align="center"> <img src=https://ai.google.dev/gemma/images/gemma4_banner.png> </div> <p align="center"> <a href="https://huggingface.co/collections/google/gemma-4" target="_blank">Hugging Face</a> | <a href="https://github.com/google-gemma" target="_blank">GitHub</a> | <a href="https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/" target="_blank">Launch Blog</a> | <a href="https://ai.google.dev/gemma/docs/core" target="_blank">Documentation</a> <br> <b>License</b>: <a href="https://ai.google.dev/gemma/docs/gemma_4_license" target="_blank">Apache 2.0</a> | <b>Authors</b>: <a href="https://deepmind.google/models/gemma/" target="_blank">Google DeepMind</a> </p>

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on small models) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.

Dense Models

Benchmark Results

Best Practices

• temperature=1.0, top_p=0.95, top_k=64
• Enable thinking: include <|think|> token in system prompt
• Place images before text in multimodal prompts
• Vision token budgets: 70, 140, 280, 560, 1120

Getting Started

from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "google/gemma-4-31B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto", device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short joke about saving RAM."},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = processor(text=text, return_tensors="pt").to(model.device)
input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=1024)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
processor.parse_response(response)

License

Apache 2.0 — see license link.