Qwen3.6-35B-A3B-Pym-Q2-MTP

Qwen3.6-35B-A3B-Pym-Q2-MTP

⚠️ Preliminary — still under active testing. The numbers below come from small-sample, in-house runs; treat them as directional. We're validating with a larger suite and will update this card (and possibly the weights) as we learn more.

Pym is SGLabs' aggressive-quantization line — shrink a large model to a fraction of its size while keeping its edge, and (where the architecture allows) its speed. This is our first release.

A ~13 GB mixed-precision GGUF quant of Qwen3.6-35B-A3B — a 256-expert hybrid SSM + attention MoE. The routed experts are taken down to 2–3 bits with an importance matrix; the attention, SSM, shared-expert, and router paths stay at Q8. Crucially, the model's native multi-token-prediction (MTP) head is preserved, so speculative decoding stays on and the quant comes out faster than the source, not just smaller.

Runs on llama.cpp — including AMD (ROCm) and Apple Silicon. No CUDA required.

The recipe

• Base: Qwen3.6-35B-A3B (256 experts / 8 active; hybrid SSM + attention; native MTP head).
• Importance matrix: computed on real agentic-coding traffic from our own workloads → ~99.6% expert coverage.
• Precision map:
  • ffn_gate_exps, ffn_up_exps → IQ2_XXS
  • ffn_down_exps → IQ3_XXS
  • attention · SSM · shared experts · router · embeddings · output · MTP head → Q8_0
• MTP preservation: upstream llama-quantize refuses to quantize the extra nextn/MTP layer under an imatrix; we patched its per-layer bounds check to allow it and forced the MTP block to Q8, keeping draft-mtp speculative decoding available on the quantized model.

Benchmarks (preliminary, in-house)

Single AMD Strix Halo (gfx1151) box, llama.cpp, vs. the Q8_K_XL source with MTP:

Read these honestly: the HumanEval gap is within sampling noise at n=30 — a lossy quant can't truly beat its own source, so the real takeaway is no measured coding-quality loss. Perplexity is about +9%, expected for a ~2-bit expert quant. The headline: ⅓ the size, ~10% faster, quality intact. A full HumanEval-164 run is in progress and this card will be updated.

Usage (llama.cpp)

# basic load (offload all layers to GPU)
llama-cli   -m Qwen3.6-35B-A3B-Pym-Q2-MTP.gguf -ngl 999 -p "Write a quicksort in Python."
llama-server -m Qwen3.6-35B-A3B-Pym-Q2-MTP.gguf -ngl 999 -c 32768

The GGUF carries a preserved MTP head — enable your llama.cpp build's MTP speculative-decoding path (draft-mtp) to get the decode speedup shown above.

Credits

Inspired by EschaLabs' Escha W2 — a beautifully aggressive quant that showed how far mixed-precision experts can go. Built independently, on our own importance matrix and tooling, in that same spirit. 🎩

Base model: Qwen3.6-35B-A3B by the Qwen team.

---

Quantized by SGLabs · maintained by @syntaxgeek