grug-27b-oQ4e-fp16

grug-27b-oQ4E-fp16

Token-efficient, quantized reasoning model based on Qwen3.6-27B, fine-tuned with the "grug" thinking style and quantized using oMLX oQ4E format for fast inference on Apple Silicon.

Model Description

grug-27b-oQ4E-fp16 is a quantized variant of ProCreations/grug-27b, itself derived from Qwen/Qwen3.6-27B. The model has been:

1. LoRA fine-tuned (r=32) on all text-stack linear layers, merged to bfloat16
2. Quantized using oMLX oQ4E (4-bit exponent-aware) at version 0.5.7
3. Converted to float16 (fp16) base dtype for ~20% faster prefill on M1/M2 Apple Silicon

The "grug" thinking style produces highly token-efficient reasoning — up to 198x fewer tokens than the base model while maintaining or improving performance on reasoning and coding benchmarks. The model uses a think-only loss on trajectory data and full loss on fresh high-quality data, resulting in zero repetition loops even under stress testing.

Architecture

Quantization Details

Why float16 over bfloat16?

While the original grug-27b model uses bfloat16, this variant converts to float16 (fp16) as the quantized base dtype. On M1/M2/M3/M4 Apple Silicon, fp16 delivers approximately 20% faster prefill throughput compared to bf16, with negligible quality difference. The quantized weights remain at ~4.8 BPW via oQ4E.

Performance Benchmarks (v2.1)

Token Efficiency

Hardware Requirements

Note: On 8 GB devices, context truncation may be required for longer sequences. For full 128K context windows, 16 GB+ unified memory is recommended.

Usage

Python (mlx-lm)

from mlx_lm import load, generate

model, tokenizer = load("your-username/grug-27b-oQ4E-fp16")
response = generate(
    model, tokenizer,
    prompt="Your prompt here",
    max_tokens=1024,
    temp=0.6,
    top_p=0.95
)
print(response)

CLI

mlx_lm.run \
  --model your-username/grug-27b-oQ4E-fp16 \
  --max-tokens 4096 \
  --temp 0.6 \
  --top-p 0.95

Recommended Inference Parameters

Training Details

• Fine-tuning method: LoRA (r=32) on all text-stack linear layers, merged to bf16
• Training data:
  • grug-think-v3-10k — agent trajectories with think-only loss
  • Fresh gpt-5.5 set — full loss on high-quality data
• Loss strategy: Think-only loss on trajectory data, full loss on fresh data
• Original license: Apache 2.0 (ProCreations/grug-27b → Qwen/Qwen3.6-27B)

License

This model card and quantization are released under the Apache License 2.0. The base model Qwen/Qwen3.6-27B and intermediate ProCreations/grug-27b are also licensed under Apache 2.0.

Citation

@misc{grug27b,
  title        = {grug-27b-oQ4E-fp16},
  author       = {Your Username},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/your-username/grug-27b-oQ4E-fp16}},
  note         = {Quantized from ProCreations/grug-27b (Apache 2.0), based on Qwen/Qwen3.6-27B}
}

Acknowledgements

• Qwen/Qwen3.6-27B by Alibaba Group / Qwen team
• ProCreations/grug-27b for the grug reasoning fine-tune
• oMLX for the oQ4E exponent-aware quantization format
• Apple Silicon community for mlx-lm inference framework

Links

• Base model: Qwen/Qwen3.6-27B
• Original grug model: ProCreations/grug-27b
• oMLX quantization: oMLX on GitHub
• mlx-lm documentation: mlx.im/docs