gemma-4-12B-it-GGUF

Gemma 4 12B IT — GGUF Quantizations

GGUF quantizations of google/gemma-4-12B-it by Google DeepMind.

Model Overview

Gemma 4 12B IT is a 12-billion parameter instruction-tuned model from Google DeepMind, built on the Gemma 4 architecture. It features:

• Reasoning — Configurable thinking modes for step-by-step problem solving
• Coding — Strong code generation, completion, and correction capabilities
• Long Context — 256K token context window
• Multilingual — Support for 140+ languages
• Function Calling — Native tool use support for agentic workflows
• License — Apache 2.0 (free to use, modify, and redistribute)

These GGUF quantizations enable running the model locally on consumer hardware using llama.cpp, Ollama, LM Studio, and other compatible tools.

Quick Start

Ollama

ollama run hf.co/liodon-ai/gemma-4-12B-it-GGUF:Q4_K_M

llama.cpp

# Install llama.cpp
brew install llama.cpp  # macOS
# or download from https://github.com/ggerganov/llama.cpp/releases

# Start server with web UI
llama-server -hf liodon-ai/gemma-4-12B-it-GGUF:Q4_K_M

# Or run directly in terminal
llama-cli -hf liodon-ai/gemma-4-12B-it-GGUF:Q4_K_M

LM Studio

1. Open LM Studio
2. Search for liodon-ai/gemma-4-12B-it-GGUF
3. Download your preferred quantization
4. Start chatting

Jan

1. Open Jan
2. Navigate to Hub
3. Search liodon-ai/gemma-4-12B-it-GGUF
4. Download and run

Available Quantizations

Hardware Requirements

Estimated VRAM requirements for full model loading (no context):

Tip: Use --cache-type-k q4_0 --cache-type-v q4_0 in llama.cpp to roughly double your available context length.

Context Length Cheat Sheet

Rough estimates for max context length at each quantization (assumes q8_0 KV cache + ~1.5 GB overhead):

Recommended Sampling Parameters

For best results with Gemma 4:

Thinking Mode

Gemma 4 supports configurable thinking modes. Enable thinking for complex tasks:

• Enable thinking: Add <|think|> token at the start of the system prompt
• Disable thinking: Remove the <|think|> token
• Default: Most libraries handle this automatically via the chat template

Model Architecture

Base Model

• Model: google/gemma-4-12B-it
• Organization: Google DeepMind
• License: Apache 2.0
• Blog: Gemma 4 Launch

Quantization Method

Quantized using llama.cpp's llama-quantize tool with the following methods:

• Q2_K — 2-bit K-quants (super-block size 16)
• Q3_K_M — 3-bit K-quants (medium quality)
• Q4_K_M — 4-bit K-quants (medium quality, recommended)
• Q5_K_M — 5-bit K-quants (medium quality)
• Q6_K — 6-bit K-quants (all tensors quantized to 6-bit)
• Q8_0 — 8-bit block-wise quantization (near-lossless)

Citation

@misc{gemma4_12b_it_gguf,
  title = {Gemma 4 12B IT GGUF Quantizations},
  author = {{liodon-ai}},
  year = {2026},
  url = {https://huggingface.co/liodon-ai/gemma-4-12B-it-GGUF},
  note = {Quantizations of google/gemma-4-12B-it by Google DeepMind}
}