gemma-4-12B-coder-fable5-composer2.5-nvfp4

gemma-4-12B-coder-fable5-composer2.5 — NVFP4 (MLX)

Original fine-tune by yuxinlu1 (gemma-4-12B-coder-fable5-composer2.5); this repo is an NVFP4 (MLX) requant for fast local inference on Apple Silicon.

A mixed-precision NVFP4 conversion of the Gemma‑4‑12B coder fine‑tune (Python/algorithmic, reasons in the thinking channel). The weights are plain MLX safetensors; built for and tested with Krill — a pure Swift + MLX inference engine for Apple Silicon (no Python, no GGUF at inference). Loads in ~1.7 s at ~6.8 GB.

The format

• Bulk weights are NVFP4 (4‑bit float, group_size 16); attention o_proj and the vision/audio projectors are kept at 8‑bit affine (group_size 64). This "protected" mixed recipe recovers the quality uniform 4‑bit loses on those sensitive modules while keeping 4‑bit speed and size.
• Conversion: bf16 safetensors → key‑remap to MLX layout → NVFP4 requant. No GGUF round‑trip.

Compatibility (please read)

This is an MLX checkpoint — not GGUF, not a HF/transformers checkpoint. To load it an engine needs (1) the gemma4_unified architecture (text+vision+audio) and (2) the mixed‑precision NVFP4 config (top‑level nvfp4 + per‑module 8‑bit overrides). Today that means Krill; it is not drop‑in for vanilla mlx_lm/mlx_vlm, and not loadable by llama.cpp/Ollama (GGUF) or transformers/vLLM.

Install Krill & run

# Homebrew:
brew tap srvsngh99/krill && brew install krill
# …or one-line installer (Apple Silicon):
curl -fsSL https://raw.githubusercontent.com/srvsngh99/Krill/main/install.sh | sh

krill pull gemma-4-12b-coder   # registered alias
krill run  srv-sngh/gemma-4-12B-coder-fable5-composer2.5-nvfp4 "Write a Python LRU cache."
krill serve --model srv-sngh/gemma-4-12B-coder-fable5-composer2.5-nvfp4 --port 57455     # OpenAI-compatible API
KRILL_ENABLE_THINKING=1 krill run srv-sngh/gemma-4-12B-coder-fable5-composer2.5-nvfp4 "..."   # reasoning channel

Benchmarks

pass@1 / accuracy, single greedy pass, run with Krill on an M4 Pro. All three models are in this same NVFP4 format (true apples‑to‑apples). HumanEval+/MBPP+ are EvalPlus (stricter tests); MBPP = the 378‑problem EvalPlus set; GSM8K = 150‑problem subset, 8‑shot. Not EvalPlus‑leaderboard‑comparable.

Takeaways: the code/agentic fine‑tunes massively out‑code the Google base on HumanEval/MBPP, while the base is stronger at math (GSM8K). Reasoning‑on helps the fine‑tunes but tends to hurt the base's coding (it over‑reasons and mangles the code block). Decode ≈ 28 tok/s.

Credits & license

Fine‑tune © its original author (yuxinlu1 (gemma-4-12B-coder-fable5-composer2.5)); base model is Google Gemma 4, under the Gemma license. This repo only changes quantization/packaging.

---

About this build

Quantized and tested by Sourav Singh for Krill, a Mac-native LLM runtime and coding agent in one Swift + MLX binary. More Apple Silicon builds: huggingface.co/srv-sngh. Notes on building with local models and frontier AI: souravailabs.ai · GitHub · LinkedIn · YouTube