Qwen3.6-35B-A3B-MLX-q3exp

Qwen3.6-35B-A3B — MLX, 3-bit experts / 4-bit everything else

▶ Try it in your browser — no install: zerotvm.com · same demo as a Space (WebGPU; ~20 GB free RAM; weights stream from this repo and cache in OPFS)

The checkpoint keeps the base model's vision tower, so mlx_lm loads it as the multimodal model it is. The browser runtime linked above is text-only — it does not build the vision path.

A mixed-precision requantization of lmstudio-community/Qwen3.6-35B-A3B-MLX-4bit (itself a 4-bit MLX build of Qwen/Qwen3.6-35B-A3B):

• expert stacks (switch_mlp.* and the shared_expert.* folded into them): 3-bit, group 64, MLX affine
• attention, gated-DeltaNet, embeddings, lm_head: 4-bit (unchanged from the source)
• router and shared-expert gate: 8-bit (unchanged)

Total: 16.36 GB (from 19.5), which is the difference between fitting and not fitting beside a browser on a 32 GB Mac.

Why

Built for zerotvm.com — browser-native inference on hand-written WebGPU kernels (?model=qwen36q3). The 4-bit build needs ~24 GB of free RAM to decode; this one ran at about 55 tok/s on a quiet 32 GB M2 Max. That is one owner session rather than a median of N, so treat it as indicative.

Experts were chosen because they are 16.2 GB of the 19.7 GB resident set and the most redundant part of a MoE.

Quality is not measured

The 3-vs-2-bit choice was made on a block-output cosine against the 4-bit block — 0.936 for 3-bit and about 0.79 for 2-bit on the layer-0 block. Two things about that number matter more than its value.

It is a fidelity measurement: how closely the 3-bit block reproduces the 4-bit block on one input. Fidelity does not predict model quality. The clearest counterexample is in the same codebase — an embedding variant with the highest cosine to the gold reference of any variant tested, 0.9379, answers 0 of 6 retrieval queries correctly.

The 0.936 figure is also not reproducible from a committed script, and the 2-bit companion figure appears as both 0.79 and 0.785 in different places.

The comparison that would answer the question is perplexity of this build against the 4-bit one over identical windows, with error bars (scripts/quality-ab.py). It has not been run. Until it has, treat 3-bit-versus-4-bit quality as unmeasured rather than acceptable.

Use with mlx_lm

config.json carries per-tensor quantization overrides, so this loads like any other MLX checkpoint:

from mlx_lm import load, generate
model, tokenizer = load("abgunaydin/Qwen3.6-35B-A3B-MLX-q3exp")
print(generate(model, tokenizer, "List the planets of the solar system.", max_tokens=64))

Provenance

Produced by scripts/convert-q3-experts.py: mx.dequantize(bits=4) → mx.quantize(bits=3) per expert tensor, everything else copied verbatim. config.json carries per-tensor quantization overrides in the same per-path style the source checkpoint uses for its 8-bit router, so mlx_lm loads this checkpoint directly.

Requantizing from the 4-bit build (rather than the bf16 original) adds only a small error on top of 3-bit's own: snapping an already-quantized value to a coarser grid mostly lands where the original would have.

Related

The engine that loads this checkpoint, and the other browser-native simulators built on the same hand-written WGSL approach:

Source: github.com/abgnydn/zero-tvm