Downloads · 30 days
75
7% of all-time downloads
aquaman164/Qwen3.6-35B-A3B-MLX-VQ-2.4bpw
Qwen3.6-35B-A3B-MLX-VQ-2.4bpw is a text generation model from aquaman164. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
Vector-quantized (trained codebooks + GPTQ compensation) MLX build of Qwen/Qwen3.6-35B-A3B — the smallest member of the VQ family, built for 16–18 GB unified memory machines. ~10.5 GB text model (+0.5 GB optional MTP…
Downloads · 30 days
75
7% of all-time downloads
All-time downloads
1.1K
Public
Parameters
5.9B
11 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors11 GB · 100%
How the weights are stored.
U323.3B · 56%
From the Hugging Face model README
Vector-quantized (trained codebooks + GPTQ compensation) MLX build of Qwen/Qwen3.6-35B-A3B — the smallest member of the VQ family, built for 16–18 GB unified memory machines. ~10.5 GB text model (+0.5 GB optional MTP head, see below); the vision tower and MTP head stay on disk for normal text use, so peak RAM is ≈ 9.8 GB + context.
⚠️ Not loadable by stock mlx-lm / LM Studio / Ollama / oMLX — use the bundled loader + Metal
kernels in code/ (pure Python, mlx>=0.31, no build step).
| build | size | think-ja | multilingual |
|---|---|---|---|
| VQ-3.4bpw | 15.02 GB | +6.0% | +5.8% |
| GPTQ-3.5bpw (scalar) | 15.35 GB | +6.6% | +6.9% |
| VQ-2.6bpw | 11.53 GB | +12.4% | +12.2% |
| GPTQ-2.7bpw (scalar) | 12.03 GB | +17.5% | +15.4% |
| this (VQ-2.4bpw) | 10.49 GB | +19.5% | +21.6% |
This is the aggressive end of the curve: quality is close to the scalar 2.7bpw build at 1.6 GB less — pick it when the machine can't fit the 11.5 GB+ builds. If you have ≥24 GB, use VQ-3.4bpw instead. (Measured with an RTN spine; the shipped artifact's spine is GPTQ-compensated, i.e. slightly better than the table.)
lm_head 6bit,
routers 8bit, embeddings 4bit. Vision tower: unquantized bf16.pip install "mlx>=0.31" "mlx-lm>=0.31"
hf download aquaman164/Qwen3.6-35B-A3B-MLX-VQ-2.4bpw --local-dir qwen-vq24
python qwen-vq24/code/vq_serve.py --model qwen-vq24 --port 8091 # OpenAI-compatible
python qwen-vq24/code/vq_generate.py qwen-vq24 100 # one-shot test
Decode uses fused Metal kernels for all three tiers (the d=8 1.5-bit path is the fastest per
dispatch): measured 73 tok/s decode on an M-series 48 GB — faster than the 2.6bpw build's
66 tok/s (fewer expert bytes + lighter d=8 index unpack). Pure-MLX reference path
(VQ_KERNEL=0): 4.1 tok/s, bit-identical outputs.
VQ_MTP=1)The artifact also ships the checkpoint's MTP head (model-mtp.safetensors, +0.5 GB:
one full-attention MoE layer, experts 4bit gs64 / attention 8bit, fc+router+norms fp16,
norms pre-shifted to the MLX +1 convention). It is inert by default — mx.load is lazy,
so RAM is unaffected. With oMLX installed in the same
venv, VQ_MTP=1 activates its mlx-lm PR#990 patches for self-speculative decoding:
| temperature | accept rate | decode |
|---|---|---|
| 0 (greedy) | 82% | 80 tok/s (+8%), output bit-identical to VQ_MTP=0 |
| 1 (sampled) | 87% | 58 tok/s (slower than baseline — hence default off) |
On a top-8 MoE the 2-token verify step reads ~2x the expert bytes of a single decode step, which caps speculative gains; the sampled path additionally pays per-cycle sampling/softmax costs. Use it for greedy workloads only.
Base model © Alibaba Cloud, Apache-2.0. Pipeline: loss-aware allocation + GPTQ-VQ encode (OneCompression); serving adapter = vqmoe model #3.