gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-MLX-OptiQ-8bit

Gemma 4 12B - Agentic & Coding (MLX 8-bit OptiQ)

An 8-bit quantized, Apple Silicon-optimized version of the agentic Gemma 4 12B model, fine-tuned for coding, terminal usage, tool-use, and multi-step reasoning. Built on top of yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 and quantized with OptiQ for higher precision inference on MLX.

---

🔍 Model Overview

---

🎯 Capabilities

✅ Coding – Writes, debugs, and explains code (Python, JavaScript, etc.) with chain-of-thought reasoning. ✅ Agentic Workflows – Uses tools, terminal commands, and multi-step reasoning (e.g., grep, ls, cat). ✅ Tool-Use – Emits structured Gemma 4 native tool calls (compatible with OpenAI-style tools API). ✅ Multimodal Ready – Supports image, audio, and video tokens (token IDs: 258880, 258881, 258884). ✅ Grounded Reasoning – Does not fabricate file paths, function signatures, or values (0% hallucination rate in testing). ✅ Higher Precision – 8-bit quantization retains more accuracy than 4-bit, ideal for tasks requiring finer detail.

---

📊 Performance

🔬 Methodology: Local evaluation, Q8_0 quant, greedy decoding, self-simulated user, 20 tasks. Not directly comparable to published leaderboards (different harness/user simulator).

✨ 8-bit vs 4-bit: Higher precision (8-bit) typically yields better accuracy for complex reasoning and coding tasks, at the cost of ~2× larger model size (~13.4 GB vs ~6.7 GB).

---

🚀 Quick Start

1. Install Dependencies

pip install mlx mlx-lm transformers

2. Load the Model (MLX)

from mlx_lm import load, generate

model, tokenizer = load("code-and-canvas/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-MLX-OptiQ-8bit")

prompt = """Write a Python function to check if a string is a valid IPv4 address."""
response = generate(model, tokenizer, prompt=prompt, temp=1.0, top_p=0.95)
print(response)

3. Chat Mode (with Thinking)

messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

# Enable thinking (default: True)
response = generate(
    model,
    tokenizer,
    prompt=tokenizer.apply_chat_template(messages, add_generation_prompt=True),
    temp=1.0,
    top_p=0.95,
    max_tokens=512
)
print(response)

4. Tool-Use (Agent Mode)

The model emits Gemma 4 native tool calls (e.g., <|tool_call>call:grep{...}<tool_call|>). Use with llama.cpp (recommended for agents):

# Requires a recent llama.cpp with gemma4_unified support
llama-server \
  -m gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-MLX-OptiQ-8bit-Q8_0.gguf \
  --jinja \
  -ngl 99 \
  --tools your_tools.json

---

---

🛠️ Configuration

Model Config (Key Settings)

Generation Config (Default)

{
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95,
  "do_sample": true,
  "bos_token_id": 2,
  "eos_token_id": [1, 106, 50],
  "pad_token_id": 0
}

Special Tokens

---

---

📁 Files

---

---

🔧 Use Cases

✅ Recommended

• High-Precision Coding – Best for complex debugging, code generation, and explanations where 8-bit accuracy matters.
• Terminal Agent – Running shell commands, grep, ls, cat, etc.
• Tool-Using Agent – Integrating with APIs, databases, or custom tools.
• Technical Q&A – Answering complex programming/CS questions with higher accuracy.
• Multimodal Tasks – Processing images/audio/video (token IDs supported).

⚠️ Considerations

💡 Choose 8-bit if:

• You need higher accuracy for complex tasks.
• You have enough VRAM/storage (~13.4 GB).
• You prioritize quality over speed.

⚠️ Limitations

• General Knowledge – Slightly weaker than base on non-technical topics (expected trade-off).
• Safety – Not safety-aligned (task-focused training). Add guardrails for production.
• Language – English-centric (other languages may underperform).

---

---

🔄 Fine-Tuning & Quantization Details

Base Model

• Original: google/gemma-4-12B-it (Google’s instruction-tuned Gemma 4).
• Fine-tune: yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2
  • v2 Improvements:
    • Agentic/Terminal: Multi-step tool-use trajectories (read → reason → act → verify).
    • Coding: Chain-of-thought over Python tasks + Fable-5-redo for hard cases.
    • General: Curated reasoning/instruction data to retain broad competence.
  • Data Sources: Fable 5 (rebuilt reasoning traces with Opus 4.8), Composer 2.5, custom agentic datasets.

Quantization

• Method: OptiQ (8-bit affine quantization).
• Group Size: 64.
• Mode: Affine.
• Compatibility: MLX (Apple Silicon), safetensors format.
• Advantage: Higher precision than 4-bit, better for nuanced reasoning.

---

---

📜 License

Apache 2.0 – Free to use, modify, and redistribute. (Inherited from Google’s Gemma 4 release.)

---

---

🙏 Acknowledgments

• Base Model: Google Gemma 4 (Apache 2.0).
• Fine-tune: yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2.
• Quantization: OptiQ (8-bit quantization for MLX).
• Framework: MLX (Apple Silicon acceleration).

---

---

💬 Example Prompts

Coding

Write a Python script to:
1. Read a CSV file.
2. Clean missing values.
3. Plot a histogram of the first numeric column.

Terminal Agent

I have a file called data.json. How do I:
1. Check if it exists?
2. Count the number of lines?
3. Extract all email addresses from it?

Tool-Use (Agent Mode)

You are a terminal assistant. Use tools to help the user.
User: List all Python files in /home/user/projects and count their lines.

Multimodal

[Image of a Python error traceback]
What is causing this error, and how do I fix it?

---

---

🐛 Issues & Support

• Bugs: Open an issue on the repository.
• Questions: Ask in Discussions.
• Contributions: PRs welcome! (Apache 2.0 compliant.)

---

---

🚀 Happy Hacking! Built with ❤️ by Code and Canvas.