Downloads ยท 30 days
32
3% of all-time downloads
liskasYR/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF
gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF is a text generation model from liskasYR. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
No matter your GPU. No matter your RAM. If you've got ~4.5 GB of VRAM or unified memory free, you can run your own private, offline AI right now. ๐ Tuned on Opus 4.6, 4.7 & 4.8 reasoning data, it delivers a major leaโฆ
Downloads ยท 30 days
32
3% of all-time downloads
All-time downloads
999
Public
Repo size
60.8 GB
Likes
0
Public
Click a slice to open those files.
.gguf36.9 GB ยท 61%
From the Hugging Face model README
No matter your GPU. No matter your RAM. If you've got ~4.5 GB of VRAM or unified memory free, you can run your own private, offline AI right now. ๐ Tuned on Opus 4.6, 4.7 & 4.8 reasoning data, it delivers a major leap in reasoning power โ whether you're asking questions or writing code. ๐ง ๐ป All local, all yours, no API, no cloud.
As of June 7, 2026, mainline llama.cpp just merged Gemma 4 MTP support โ so the MTP draft model is now
live in the MTP/ folder.
Drop it next to any quant and generation gets noticeably faster with identical output (speculative decoding is
lossless) โ just add a couple of flags. ๐ See โก Speed it up with MTP below. ๐
| Quant | Size | Vibe |
|---|---|---|
| ๐ข Q2_K | 4.5 GB | tiniest โ runs almost anywhere |
| ๐ต Q4_K_M | 6.87 GB | the sweet spot ๐ (recommended) |
| ๐ฃ Q6_K | 9.11 GB | near-lossless |
| โช Q8_0 | 11.8 GB | basically full quality |
| (f16) | 22.2 GB | full precision (overkill for most) |
Rough estimates ๐ค (assumes q8_0 KV cache + ~1.5 GB overhead; use q4_0 KV cache for โ2ร more context!).
Max context is 131K. "โ" = won't fit, pick a smaller quant. โ๏ธ
| Your VRAM / unified mem | ๐ข Q2_K (4.5G) | ๐ต Q4_K_M (6.87G) | ๐ฃ Q6_K (9.11G) | โช Q8_0 (11.8G) |
|---|---|---|---|---|
| 8 GB | ~16K ctx | tight (~2โ4K) | โ | โ |
| 12 GB | ~48K | ~30K | ~12K | โ |
| 16 GB | ~80K | ~64K | ~44K | ~22K |
| 24 GB | 131K (max) ๐ | ~128K | ~110K | ~88K |
| 32 GB | 131K | 131K | 131K | 131K |
๐ก Apple Silicon / integrated GPUs with unified memory count too โ same numbers, just slower than a dGPU. ๐ก Low on room? Drop a quant or switch KV cache to
q4_0and your context roughly doubles.
New as of June 7, 2026! Gemma 4's Multi-Token Prediction drafter lets the model guess a few tokens ahead and verify them in one shot โ so you get more tokens/sec with byte-for-byte identical output. Pure speed, zero quality cost. ๐ช
1. Grab the tiny draft from the MTP/ folder:
| Draft file | Size | Use it for |
|---|---|---|
โช gemma-4-12B-it-MTP-Q8_0.gguf | 0.44 GB | recommended โ tiny + full speed |
โฆ-F16.gguf / โฆ-BF16.gguf | 0.82 GB | full-precision draft (overkill) |
๐ก The draft is tiny โ keep it Q8 or higher (over-quantizing a draft just lowers its hit rate). It pairs with any quant of the main model.
2. You need a fresh llama.cpp build โ June 7 2026 (b9553) or newer. MTP was just merged, so older builds
can't load the draft (unknown architecture: 'gemma4-assistant').
3. Run it exactly like below, just +3 flags (--model-draft, --spec-type, --n-gpu-layers-draft):
@echo off
cd /d C:\llama.cpp
llama-server.exe ^
-m C:\models\gemma4-opus48-Q4_K_M.gguf ^
--model-draft C:\models\MTP\gemma-4-12B-it-MTP-Q8_0.gguf ^
--spec-type draft-mtp --spec-draft-n-max 4 ^
--ctx-size 16384 --n-gpu-layers 99 --n-gpu-layers-draft 99 ^
--no-mmap -fa on ^
--temp 1.0 --top-p 0.95 --top-k 64 ^
--host 0.0.0.0 --port 18080
pause
Measured on a single RTX 5090 (Q4_K_M main + Q8 draft): ~1.3ร faster at greedy and ~1.2ร at the default thinking sampling โ free, with no change to output. ๐
๐ง Heads-up: this is the stock Gemma drafter (trained on base Gemma 4), so on this fine-tune the hit rate โ and thus the speedup โ is a little lower than on vanilla Gemma 4. A re-aligned draft could push it higher (maybe a future update). Either way: free speed, no downside. ๐
โฆ-Q4_K_M.gguf) and llama-server from llama.cpp.
โ ๏ธ Needs a recent llama.cpp (this is the
gemma4_unifiedarchitecture โ older builds won't load it).
.bat shown โ tweak --port, --ctx-size to taste):@echo off
cd /d C:\llama.cpp
llama-server.exe ^
-m C:\models\gemma4-opus48-Q4_K_M.gguf ^
--ctx-size 16384 ^
--n-gpu-layers 99 ^
--no-mmap ^
-fa on ^
--cache-type-k q8_0 --cache-type-v q8_0 ^
--temp 1.0 --top-p 0.95 --top-k 64 ^
--host 0.0.0.0 --port 18080
pause
http://localhost:18080 and chat. ๐ (Tip: bump --ctx-size per the table; use q4_0 KV for more.)Works in LM Studio, Jan, Ollama, etc. โ just import the GGUF, pick your quant, go. ๐พ
This model thinks in Gemma's native thought channel. Keep enable_thinking=true (the default chat template
handles it). Recommended sampling: temp 1.0, top_p 0.95, top_k 64.
google/gemma-4-12B-it.angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k,
augmented with additional Opus 4.8-generated reasoning samples I curated and mixed in.