Qwen3.6-35B-A3B-Alis-MLX-Dynamic

Qwen3.6-35B-A3B-Alis-MLX-Dynamic

Apple Silicon (MLX) mixed-precision quantizations of Qwen/Qwen3.6-35B-A3B — a 35B MoE with 3B active parameters, 3:1 Gated DeltaNet : full attention hybrid (qwen3_5_moe). One repo, one card — each quantization lives on its own branch.

* single-stream decode on an M3 Ultra, short prompt. A3B routing makes this the fastest model class per GB in the lineup — ~3× the decode rate of the dense 27B sibling.

Everything below is measured, not projected — KL / top-1 flip on a fixed 4096-token EN/code/ZH/KO slice (teacher-forced vs bf16), strided perplexity (ctx 2048 / stride 1024), and single-stream decode on an M3 Ultra.

Which build?

• 24 GB+ Mac → main. 16 GiB weights, 17.5 GB measured peak at short context, 102 tok/s.

• 16 GB Mac → 16gb. 11 GiB weights, 11.8 GB measured peak — exceeds the default ~10.7 GiB wired limit; raise it first:

  sudo sysctl iogpu.wired_limit_mb=13312
  

  With int8 KV that leaves ~1.5 GiB of cache ≈ 150 K tokens of context (KV here is tiny — see below). 111 tok/s decode makes this the best speed-per-GB option for 16 GB machines; prefer it over the dense 27B 16 GB build when throughput matters more than per-token quality.

Multimodal (default)

main carries the full multimodal stack — 4.0bpw text weights + the vision tower at 8-bit (verified with EN/KO OCR tests). Plain mlx-lm loads main fine for text-only use; the text branch saves 0.60 GB if you never need images.

pip install mlx-vlm
python -m mlx_vlm generate --model avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic \
  --image photo.jpg --prompt "이미지의 텍스트를 읽어줘." --max-tokens 300

Usage

from mlx_lm import load, generate

model, tokenizer = load("avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic")                  # main
model, tokenizer = load("avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic", revision="16gb")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "MoE 라우팅을 두 문장으로 설명해줘."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

CLI / server:

mlx_lm.generate --model avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic --revision 16gb --local-dir qwen36-35b-16gb
mlx_lm.server --model ./qwen36-35b-16gb --port 8080

Long context: --kv-bits 8 --kv-group-size 64 in the generate path (measured quality-free, below).

⚠️ 16gb under review. The dense-27B sibling's same-calibration low-bit tier was withdrawn after free-running generation showed early stops / repetition loops on long-CoT problems (teacher-forced KL/PPL did not catch it). This branch shares that calibration recipe and is being re-verified generatively; treat it as experimental until this notice is removed.

Recipes (verified from each build's config.json)

92% of the parameters live in 256 fused routed experts per MoE layer (stored as 3D SwitchLinear tensors — the sensitivity scan samples 8 experts/layer/matrix). Per-tensor bits assigned by marginal utility from a 2–6-bit + nvfp4 scan of all 472 text tensors:

nvfp4 vs affine — measured both ways. On the dense 27B sibling, affine-only beat nvfp4 decisively (its card has the numbers). Here the reverse holds: an affine-only 4.0bpw rebuild (50.3% a3 + 49.4% a4) scored KL 0.309 vs main's nvfp4-mix 0.292 — nvfp4 g16 genuinely suits the small fused expert matrices (2048×512). The format choice is architecture-dependent; main stands as-is.

DWQ. Layerwise DWQ (KL distillation vs bf16, 260-sample EN/KO/ZH/code chat mix, seq 512, validation-gated — alis-dwq): at 4.0bpw every round was rejected (already teacher-close), so main carries conversion-time weights; at 2.7bpw DWQ improved valid KL 0.305 → 0.268 (−12%).

Quality

KL(bf16‖quant) / top-1 flip on the fixed 4096-token slice, per domain:

Strided perplexity (ctx 2048, stride 1024; wikitext / code / Korean):

Generative benchmark (GPQA-Diamond, our harness, n=198, temp 1.0, 32K cap)

main scores 76.3% (avg 14,742 thinking tokens, 7 no-answer). For scale: a dense 27B at 4.6bpw scores 72.7 on the same harness, and quantized short-thinking models reach 83.3 — see the cross-family curve on the ThinkingCap repo. The 16gb 2.7bpw tier fails hard reasoning generatively (MMLU-300 75.3% but GPQA non-convergent) — hence the warning above; treat it as a light/general-use build only.

KV cache — measured recommendations

Only 10 of 40 layers carry KV (the rest are Gated DeltaNet recurrent state), so KV is tiny to begin with; quantizing it is still measured quality-free:

Notes & provenance

• Text-only conversion: the base checkpoint's vision tower and MTP head are not included (mlx-lm text path). An mtp-bf16 sidecar branch may land later.
• Calibration/eval data: tulu-3 SFT chat (EN), Korean chat, code/ZH mix — chat-template rendered.
• Built on an M3 Ultra 512 GB · mlx-lm 0.31.3 · mlx 0.31.2 · 2026-07.
• Sibling repo: Qwen3.6-27B-Alis-MLX-Dynamic (dense, higher per-token quality; its 16gb-thinkingcap branch fits 16 GB Macs without sysctl).

Credits

• Qwen team — the Qwen3.6-35B-A3B base model and its hybrid DeltaNet MoE architecture.
• Apple MLX team — mlx / mlx-lm, including the DWQ trainer this pipeline builds on.
• Unsloth — their NVFP4 releases motivated the diverse-chat calibration mix and served as quality reference points.
• AllenAI — tulu-3 SFT mixture used in the calibration set.
• Quantization recipe, DWQ retune and evals: Alis (avlp12) with the alis-dwq pipeline; built with assistance from Claude (Anthropic).