Grug-12B-ROCmFP4_FAST-GGUF

Grug-12B ROCmFP4_FAST — GGUF

ROCmFP4_FAST quant of kai-os/Grug-12B — a compact-reasoning fine-tune of google/gemma-4-12B-it, produced via charlie12345/ROCmFPX. Benchmarked on an RDNA4 RX 9060 XT (16 GB). Mean AEON score: 0.558 (120 scored / 150 cases).

Grug-12B is a specialized reasoning model trained to compress chain-of-thought while maintaining reasoning quality. At 6.0 GB, it leaves >7 GB VRAM headroom even at 152k context — by far the most comfortable quant on 16 GB cards.

File

NOT a stock llama.cpp quant

ROCmFP4_FAST is a custom weight format from charlie12345/ROCmFPX. Stock llama.cpp will exit with unknown quantization at load time. The system_fingerprint of a correctly-served ROCmFPX GGUF is b1-11d76c2.

Scope of these benchmarks — read this first

These numbers are a light baseline, not a thorough ROCmFPX evaluation. The mesh's bench framework is built for production agent workload regression-detection on the local stack, not for the kind of multi-axis sweep that upstream quant maintainers typically publish. Specifically:

• Harness scope is bounded. The numbers below come from the mesh's Aeon-Bench-Pod (self-reported mode, 150-case full suite, text-only). That's a regression suite, not a quality benchmark.
• Sample sizes are small. 150 cases on a single GPU, single rep. None are powered for statistical significance.
• No perplexity / wikitext / MMLU / GSM8K. The mesh's stack isn't a quality benchmark — those are upstream's territory.
• Single GPU class (RDNA4 16 GB). All measurements are on an AMD RX 9060 XT 16 GB (gfx1200, ROCm 7.x). No Blackwell, no CDNA, no multi-GPU. Cross-hardware generalization is NOT implied.
• No human eval. "0.558 mean on the AEON suite" is not a quality verdict on this specific quant.
• 30 prose cases unscored — no frontier judge endpoint configured, so the prose category is excluded from the mean.

What this IS good for: a quick signal that the quant (a) loads, (b) runs at good throughput on RDNA4, (c) has ample VRAM headroom, (d) produces coherent output across math, instruction, reasoning, and coding. What this is NOT good for: claiming "this is the best quant of this model," reproducing academic benchmark results, or substituting for upstream's validation work.

For a rigorous view, see kai-os/Grug-12B (parent model) and charlie12345/ROCmFPX (quantizer). Raw bench reports are attached as BENCH-*.md files in this repo.

What we measured

AEON v3 — 150 cases (120 scored)

Benchmarked on AMD RDNA4 RX 9060 XT 16 GB, full suite.

Profile: Grug excels at reasoning (0.767) and coding (0.800), consistent with its compact-reasoning training. Weak on math (0.233) — the compression training may penalize arithmetic precision. Strong difficulty scaling: holds up well at frontier (0.396) compared to other small quants.

Quick start

# Build charlie12345/ROCmFPX
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
mkdir build && cd build
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1200
make -j$(nproc)

# Serve
llama-server \
  -m Grug-12B-ROCmFP4_FAST.gguf \
  --host 0.0.0.0 --port 8081 \
  -ngl 99 -c 32768 \
  -ctk q4_0 -ctv q4_0 \
  -fa on --cache-ram 0 \
  -np 1 --batch-size 512 --ubatch-size 128 \
  --jinja --metrics -rea off

Do NOT set GGML_HIP_ENABLE_UNIFIED_MEMORY=1 on discrete AMD cards — it moves the entire model to system RAM (30× regression).

Reproduce the quant

# From kai-os/Grug-12B safetensors, converted to BF16 GGUF first:
python3 convert_hf_to_gguf.py /path/to/kai-os/Grug-12B --outfile Grug-12B-BF16.gguf

# Then quantize:
llama-quantize --allow-requantize Grug-12B-BF16.gguf \
  Grug-12B-ROCmFP4_FAST.gguf Q4_0_ROCMFP4_FAST

Files in this repo

What's NOT in this repo (caveats)

• Stock llama.cpp will not load this file. ROCmFP4_FAST is a custom weight format from charlie12345/ROCmFPX.
• Grug is NOT a general-purpose instruct model. It's a specialized compact-reasoning model. Use for math, code reasoning, and agentic pipelines where reasoning-token compression matters. Do NOT use for instruction-following tasks where clean output is expected — the reasoning bloat will hurt quality.
• No CUDA / non-AMD GPU bench. All measurements are RDNA4 (gfx1200).
• No quality benchmark (perplexity, MMLU, GSM8K). Compact-reasoning fine-tunes may not reflect academic benchmarks.
• No MTP / speculative-decode bench. Not applicable on this architecture.

Provenance

• Source model: kai-os/Grug-12B (license: other — verify commercial use terms)
• Base model: google/gemma-4-12B-it (Gemma 4 license)
• Conversion: convert_hf_to_gguf.py (BF16 GGUF from 7 sharded safetensors, 23.8 GB → ROCmFP4_FAST 6.0 GB)
• Quantizer: charlie12345/ROCmFPX commit 5b39566, preset Q4_0_ROCMFP4_FAST
• Build hardware: Node B — RDNA4 RX 9060 XT 16 GB (gfx1201), ROCm 7.2.3, NixOS
• Bench harness: Aeon-Bench-Pod v1 (self-reported mode)

License

This quant is derived from kai-os/Grug-12B (license: other — verify commercial use is permitted before deployment), which is a fine-tune of google/gemma-4-12B-it (Gemma 4 license). The ROCmFP4 quant format is provided by charlie12345/ROCmFPX (MIT).