grug-9b-qat-q4

grug-9b-qat-q4

grug put 9b bird in four-bit cave during training. bird feel rounding rock before final squish. then grug make real Q4_K_M GGUF. this not normal squish. this QAT recovery.

QAT start from full grug-9b, never train old GGUF, never requantize quant rock. grug fake-quant text linear weight as asymmetric int4, group 32, straight-through gradient. train 3,628,102 token from grug-think. full QAT overcorrected, so grug keep 25% QAT move and 75% original anchor. fixed eval gate choose before release. then export fresh BF16 GGUF and quantize one time to Q4_K_M.

numbers. same cave, same harness

both rocks run same llama.cpp build, greedy, same prompt. control = ordinary grug-9b-Q4_K_M. QAT = this rock.

grug reasoning format stay same: short grug brain inside <think>...</think>, normal say-word outside, XML tool call untouched. grug measure presence and length above, not merely hope.

grug honest: QAT help model adapt to 4-bit rounding. QAT not magic promise. numbers above say what happen on these probe. hardest real repo task may act different.

rock

grug-9b-qat-Q4_K_M.gguf — about 5.6 GB. good default for 8 GB VRAM cave.

llama-cli -hf ProCreations/grug-9b-qat-q4:Q4_K_M

need recent llama.cpp with qwen3_5 support. thinking live in <think> tag and stay short on purpose. tool call use model XML format.

recipe. grug show work

• source: ProCreations/grug-9b
• data: ProCreations/grug-think
• QAT: full text-stack linear weights, fake int4 asymmetric group size 32
• training: 684 optimizer step, 3,628,102 token, LR 2e-06
• release blend: 25% QAT checkpoint + 75% original checkpoint. grug show this because honest rock better than secret rock
• target export: llama.cpp Q4_K_M, quantized from fresh BF16 conversion
• vision tower not in GGUF. text bird only

training config and measured JSON live in this repo. grug no hide recipe behind bush.