Grug-12B-GGUF

Grug 12B GGUF

This is the adjacent GGUF release for kai-os/Grug-12B. The main repo contains the merged Transformers/safetensors fine-tune; this repo contains a llama.cpp quantized file for local inference.

Files

• Grug-12B-Q4_K_M.gguf - practical mixed-precision 4-bit GGUF quant, about 7.0 GB.
• banner.png - model banner.

Conversion

Converted from the merged full model release, then quantized with llama.cpp.

• Source model: kai-os/Grug-12B
• llama.cpp commit: 4fc4ec5
• Conversion outtype: BF16
• Uploaded quant: Q4_K_M
• Quantized size: 7024.34 MiB
• Bits per weight: 4.95 BPW
• SHA256: 3928e9af604369c111ec7098660781f26e3dc350080e3786ef9dd69881967348

Usage

Use a recent llama.cpp build with Gemma 4 / Gemma4 Unified GGUF support.

hf download kai-os/Grug-12B-GGUF Grug-12B-Q4_K_M.gguf
llama-cli -m Grug-12B-Q4_K_M.gguf -p "What is 2+2? Answer briefly." -n 64

Training Summary

Grug 12B is a compact-reasoning fine-tune of google/gemma-4-12B-it trained with QLoRA, then merged into the base model for release. The training target is terse, high-density reasoning that preserves constraints, branching decisions, invariants, edge cases, and final-answer checks while reducing unnecessary reasoning-token verbosity.

See the full model card at kai-os/Grug-12B for the technique details, dataset provenance, filtering notes, and benchmark notes.

Thanks to Lambda for the compute credits used for this work.