gemma-4-26b-a4b-opus-4.7-distilled-gguf

Gemma-4-26B-A4B-IT — Claude Opus 4.6/4.7 Reasoning Fine-tune · GGUF (Unsloth)

GGUF (llama.cpp) quantizations of a fine-tune of google/gemma-4-26B-A4B-it (via the Unsloth-fixed checkpoint unsloth/gemma-4-26b-a4b-it), trained on angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k — a ~8.7k-example reasoning trace dataset distilled from Claude Opus 4.6 / 4.7.

These files are designed for CPU / GPU inference with llama.cpp and downstream runtimes (Ollama, LM Studio, GPT4All, KoboldCpp, text‑generation‑webui, llama-cpp-python, etc.) and include a separate multimodal projector (mmproj) so the vision tower can be loaded for image inputs.

Quantized with Unsloth from the bf16 fine-tune. See the original (unquantized) HF Transformers weights for full multimodal (audio + video) support.

---

Available Quants

All quants are derived from the same bf16 fine-tune. Pair any text quant with the mmproj file to enable image input.

Note: this is a sparse MoE (~26 B total / ~4 B active per token). Memory footprint is dominated by the stored experts (all 128), so file sizes scale like a 26 B dense model — but inference compute scales like a 4 B model since only the top-8 experts run per token.

Sizing guide

---

Model Summary

---

How to Run

Requires a build of llama.cpp (or downstream runtime) with Gemma 4 support. Older builds will fail to load the GGUF.

llama.cpp — text only

# Build (one-time)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # or -DGGML_METAL=ON on macOS
cmake --build build -j

# Interactive chat
./build/bin/llama-cli \
  -m gemma-4-26b-a4b-it.Q4_K_M.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  -cnv

llama.cpp — multimodal (image input)

Pair any text quant with the BF16-mmproj.gguf companion:

./build/bin/llama-mtmd-cli \
  -m       gemma-4-26b-a4b-it.Q4_K_M.gguf \
  --mmproj gemma-4-26b-a4b-it.BF16-mmproj.gguf \
  -c 8192 -ngl 99 \
  --image path/to/picture.png \
  -p "Describe the picture and reason about anything unusual."

OpenAI-compatible server

./build/bin/llama-server \
  -m       gemma-4-26b-a4b-it.Q4_K_M.gguf \
  --mmproj gemma-4-26b-a4b-it.BF16-mmproj.gguf \
  -c 8192 -ngl 99 \
  --host 0.0.0.0 --port 8080 \
  --jinja

--jinja makes the server use the model's embedded Gemma‑4 chat template (with the <|channel>thought reasoning channel and native tool-calling).

Then point any OpenAI client at http://localhost:8080/v1.

Ollama

# Create a Modelfile
cat > Modelfile <<'EOF'
FROM ./gemma-4-26b-a4b-it.Q4_K_M.gguf

# Multimodal projector
ADAPTER ./gemma-4-26b-a4b-it.BF16-mmproj.gguf

PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 64
PARAMETER num_ctx 8192
EOF

ollama create gemma4-26b-a4b-reasoning -f Modelfile
ollama run gemma4-26b-a4b-reasoning

LM Studio / Jan / GPT4All / KoboldCpp / text-generation-webui

Drop the chosen *.gguf (and the BF16-mmproj.gguf if you want image support) into the app's model directory. Make sure the runtime ships a llama.cpp build with Gemma 4 support.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path  = "gemma-4-26b-a4b-it.Q4_K_M.gguf",
    n_ctx       = 8192,
    n_gpu_layers= -1,
    chat_format = "gemma",        # uses embedded Jinja template
)

resp = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a careful, step-by-step reasoner."},
        {"role": "user",   "content": "If a train leaves at 9:15 and travels for 2h 47m, when does it arrive?"},
    ],
    temperature=1.0, top_p=0.95, top_k=64,
    max_tokens=1024,
)
print(resp["choices"][0]["message"]["content"])

---

Recommended Sampling

From the source generation_config.json:

For deterministic reasoning, drop temperature to ~0.3–0.6.

---

Chat Template & Reasoning Channel

The Gemma-4 chat template is embedded in every GGUF, and supports:

• Role turns: <|turn>system|user|model<turn|>
• Thinking channel: <|channel>thought ... <channel|> — enabled when the template receives enable_thinking=true (llama-server: send "chat_template_kwargs": {"enable_thinking": true} in the request, or use /v1/chat/completions with "reasoning_effort": "high" on a compatible client).
• Tool declarations: <|tool>…<tool|>
• Tool calls / responses: <|tool_call>…<tool_call|> / <|tool_response>…<tool_response|>
• Multimodal placeholders: <|image|>, <|audio|>, <|video|> (only <|image|> is wired up in llama.cpp via the mmproj).

When generation starts without enable_thinking, the template emits an empty <|channel>thought<channel|> block to suppress reasoning. Pass enable_thinking=true to unlock the model's full chain-of-thought.

---

Training

The training corpus emphasizes:

• Long, structured chain-of-thought reasoning
• Math, code, logic and step-wise problem decomposition
• Self-verification and answer revision patterns
• Instruction following with explicit thinking → answer separation

Reasoning data is distilled from Anthropic's Claude models. Outputs may reflect stylistic patterns of Claude (e.g. hedged tone, explicit step labels, "Let me think…" preambles).

---

Intended Use

Primary use cases

• Local reasoning-heavy assistants (math, coding, agentic planning) on commodity GPUs / Apple Silicon
• Multimodal Q&A over images
• Long-context summarization, retrieval, and document analysis
• Tool-calling / function-calling agents (template-native)
• Edge / offline deployments where bf16 weights are too large

Out-of-scope / not recommended

• High-stakes decisions (medical, legal, financial advice) without human review
• Generation of disallowed content under the Gemma Prohibited Use Policy
• Safety-critical autonomous deployments without guardrails

---

Files

---

Limitations & Biases

• Quantization loss: Lower-bit quants (Q2_K_L, Q3_K_M) will degrade reasoning quality, especially on long chains of thought. Prefer Q4_K_M or higher for reasoning tasks.
• MoE quirks: K-quant kernels for MoE experts are still being optimized in llama.cpp; performance and quality may improve in newer builds.
• Multimodal scope in GGUF: Only image input is supported via mmproj. Audio and video inputs require the original Transformers checkpoint.
• Hallucinations: Like all LLMs, the model can produce confident but incorrect answers, especially outside its training distribution.
• Reasoning style transfer: Because SFT data is distilled from Claude, stylistic and refusal patterns may leak into outputs.
• Dataset size: ~8.7k examples is small; expect a targeted style/reasoning shift rather than broad capability uplift.
• Safety: No additional safety fine-tuning was performed. Base Gemma-4 safety guarantees apply; add your own guardrails for production.

---

License

• This fine-tune: GPL-3.0
• Base model: subject to the Gemma Terms of Use and Gemma Prohibited Use Policy. You must comply with both when using or redistributing this model.
• Training data: see the dataset card for terms.

---

Citation

@misc{gemma4_2025,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2025},
  url    = {https://ai.google.dev/gemma}
}

@misc{unsloth,
  title  = {Unsloth: 2x faster LLM fine-tuning with 70% less memory},
  author = {Daniel Han and Michael Han and {Unsloth team}},
  year   = {2024-2026},
  url    = {https://github.com/unslothai/unsloth}
}

@misc{llama_cpp,
  title  = {llama.cpp},
  author = {Georgi Gerganov and contributors},
  year   = {2023-2026},
  url    = {https://github.com/ggml-org/llama.cpp}
}

@misc{claude_reasoning_8k7,
  title  = {claude-opus-4.6-4.7-reasoning-8.7k},
  author = {angrygiraffe},
  year   = {2026},
  url    = {https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k}
}

---

Acknowledgements

• Google DeepMind — Gemma-4 base model
• Unsloth team — Quant-fixed checkpoint, training framework, and GGUF quantization
• Georgi Gerganov & llama.cpp contributors — GGUF format and inference runtime
• angrygiraffe — Reasoning distillation dataset
• Anthropic — Source model family (Claude Opus 4.6 / 4.7) for the distilled reasoning traces