Grug-35B-A3B-MLX

Grug-35B-A3B MLX

Apple Silicon MLX quantizations of kai-os/Grug-35B-A3B, packaged as a single Hugging Face repo with one folder per quantization level.

This repository contains MLX safetensors folders, not GGUF files. The source model remains in the upstream Hugging Face repository.

Available variants

Local benchmark notes

Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB unified memory using oMLX.

4-bit

Screenshot: benchmarks/grug-35b-a3b-4bit-omlx.png

Continuous batching, pp1024 / tg128:

6-bit

Screenshot: benchmarks/grug-35b-a3b-6bit-omlx.png

The 6-bit batching run shown in benchmarks/grug-35b-a3b-6bit-prefill-cap.png hit oMLX's prefill safety cap at the tested settings: predicted peak exceeded 45.5 GB, which was 90% of the effective 50.5 GB ceiling.

8-bit

The 8-bit variant could not be thoroughly tested under the default local memory cap. In the initial oMLX test environment, oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning.

The 8-bit variant should be retested after raising the Apple GPU wired-memory cap and restarting the local serving process, for example:

sudo sysctl iogpu.wired_limit_mb=59392

Throughput numbers are not published yet. This section will be updated after a successful full benchmark run.

Usage

Download only the variant you want:

from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)

Run with mlx-lm:

python -m mlx_lm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Reply with a short explanation of what this model is." \
  --max-tokens 256

Local compatibility note

The source config uses model_type: qwen3_5_moe_text. At conversion time, mlx-lm 0.31.3 did not include a native loader for that exact model type, so the conversion used a local compatibility shim mapping the model to the existing Qwen3-Next style MLX implementation and adapting the published weight names.

Consumers may need an mlx-lm build that supports qwen3_5_moe_text, or an equivalent local compatibility shim, until upstream support is available.

Provenance and attribution

• Source model: kai-os/Grug-35B-A3B
• Relationship: MLX quantized derivatives of the source model
• Source revision used locally: f6349f9c7beba10ad44ce0210b0a0f6fba414a05
• Conversion tool: mlx-lm 0.31.3
• Quantization mode: affine, group size 64

Please refer to the source model card for upstream training details, intended use, limitations, acknowledgements, and license context.

Limitations

Quantization can change output quality, numerical behavior, and edge-case performance. These files are intended for local MLX inference on Apple Silicon. Use the source model repo for the original Transformers/safetensors weights.