grug-35b-v2-6bit-XL-mlx

leonsarmiento/grug-35b-v2-6bit-XL-mlx

grug big brain. grug think fast, talk less, get job done. now grug shrink for apple metal — BaseQuant_XL 6/8-bit recipe. router stay raw bf16, expert muscle get 6-bit gym. vision eye still work.

grug-35b-v2 built by ProCreations on top of Ornith-1.0-35B (Qwen3.5 MoE, 35B total / ~3B active). LoRA on attention + DeltaNet + shared paths only — router and experts untouched. v2.1 added long-hunt data, deep-think, stuck-loop escape. Result: grug think in dense grug-speak inside <think>, answer normal english. Old v1 repetition sickness: extinct.

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

how run

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/grug-35b-v2-6bit-XL-mlx --max-tokens 256 --temperature 0.6 --top-p 0.95 --prompt "Hello"

grug prefer --reasoning-parser deepseek_r1 in server. LM Studio work too — vision mmproj included.

Quantization Strategy

• Bits per weight: 6.808 · Total size: ~28 GB (6 shards) · Group size: 64

Recommended Inference Parameters

MLX Benchmarks (n=50, instruct mode)

Full comparison against 9 local SOTA MoE models in the Local SOTA for 48GB Macs collection.

grug strong on knowledge (MMLU 80%, +16 over Ornith base) and reasoning (MMLU_PRO tied with Ornith at 66%). Coding trails the Gemma-4 champion as expected — grug is a generalist conversational reasoner, not a coding specialist.

Thinking-Mode MATHQA: Reasoning Efficiency (n=10, seed=42)

Same 10 MATHQA questions, thinking mode, 8192-token budget, temp=1.0. 3-stage protocol: initial run → 5-pattern answer extraction → re-test all failures.

grug is the token-efficiency champion — 13× fewer reasoning characters per correct answer than the next best model (Gemma-4). On the hardest question (Q636, probability), grug used 220 chars to reason where Ornith used 5,682 and Gemma used 7,085. grug got it wrong, but the reasoning was dense, not lazy.

Failure Analysis (after re-test)

The one genuine failure (Q636) is a probability word problem — grug's compressed reasoning style trades depth for speed, occasionally missing subtle constraint interactions. The parsing failure (Q2872) reveals a side-effect of extreme token efficiency: with only ~150 chars of reasoning, the answer format can be ambiguous to regex extraction.

Why grug Wins on Efficiency

ProCreations trained grug to think in dense "grug-speak" inside <think> blocks — eliminating filler, restatement, and self-verification loops. Other 35B MoE models produce 2,000–8,000 reasoning chars per question; grug averages 185. This translates to:

• 15× faster wall time per correct answer (4.2s vs 63.8s for Ornith)
• Context window preservation — less reasoning = more room for the actual conversation
• Energy savings — fewer generated tokens = less compute per answer

The trade-off: 90% accuracy vs 100% for Qwen3.6/Gemma/Ornith. For the 1 question grug genuinely missed, verbose reasoning from the other models would have caught the error.

Source Model

Source benchmarks: HumanEval 84.8, MBPP 88.0, GSM8K 92.0, MATH-500 64.7. Highly token-efficient — 93–97% token reduction vs base model across benchmarks. Zero degenerate loops, 100% think blocks closed.

grug made by ProCreations. Base brain by DeepReinforce. Metal shrink by leonsarmiento.