gemma-4-12B-it-GGUF

Gemma 4 12B IT - GGUF

This repository contains highly optimized GGUF format quantized weights for Google's Gemma 4 12B IT.

These weights were compiled using the bleeding-edge master branch of llama.cpp to ensure full architectural compatibility and natively support high-speed inference on modern hardware. Gemma 4 12B is an 11.95 billion parameter dense, open-weights model. It features a unified, encoder-free architecture that directly processes raw image patches and audio waveforms alongside text without requiring separate vision or audio encoders. This model natively supports a 256K context window, a built-in thinking/reasoning mode, and tool-use capabilities for function calling.

📦 Available Quantization Formats

Below are multiple quantization "flavors" to best match your hardware capabilities.

Note: Context windows (e.g., 8K or the maximum 256K) will require additional VRAM allocation for the KV Cache.

🚀 How to Use

These GGUF files are designed to be run locally using llama.cpp or any compatible downstream UI (e.g., LM Studio, Ollama, or custom Gradio WebUIs).

💻 Command Line (llama.cpp)

To run the recommended Q4_K_M model with full GPU offloading and an 8K context window:

./llama-cli -m gemma-4-12b-it-Q4_K_M.gguf -n 2048 -c 8192 -ngl 999 -p "You are an expert AI assistant. Explain quantum entanglement."

🐍 Python (llama-cpp-python)

from llama_cpp import Llama

# Load the model with Blackwell-optimized Flash Attention
llm = Llama(
    model_path="./gemma-4-12b-it-Q4_K_M.gguf",
    n_gpu_layers=-1, # Offload entirely to GPU
    n_ctx=8192,      # 8K Context Window
    flash_attn=True
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a python script to calculate the Fibonacci sequence."}
    ]
)

print(response["choices"][0]["message"]["content"])

⚖️ License & Acknowledgements

These weights are derivative works of Google's Gemma 4 12B IT model. The original model is distributed under the Apache 2.0 License. All credit for the underlying neural architecture and base training data goes to the Google DeepMind team.