Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8

Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8

MLX mixed 4-/8-bit quantization of tvall43/Qwen3.6-35B-A3B-heretic, targeting Apple Silicon via mlx-lm.

Text-only. Vision-tower weights are not present in this quantization (the base model's vision encoder was excluded during conversion). Despite config.json still declaring Qwen3_5MoeForConditionalGeneration (required for mlx-lm compatibility), image inputs will fail. For multimodal use, see the original Qwen/Qwen3.6-35B-A3B.

~20 GB on disk. Runs on Apple Silicon Macs with 64 GB unified memory for comfortable generation.

---

Lineage

Qwen/Qwen3.6-35B-A3B
  └─ tvall43/Qwen3.6-35B-A3B-heretic   (abliterated)
       └─ cspenn/Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8   (this repo, MLX quant)

Qwen/Qwen3.6-35B-A3B — Qwen's open-weight 35B MoE model with only 3B parameters activated per token. Features 40 hybrid transformer layers (alternating Gated DeltaNet linear-attention and Gated Attention blocks), 256 routed experts + 1 shared expert, and a native context window of 262,144 tokens. Notable for strong agentic/coding performance and multimodal capability.

tvall43/Qwen3.6-35B-A3B-heretic — A decensored derivative produced with the Heretic v1.2.0 abliteration technique. Safety refusals were reduced from ~86/100 to ~5/100 by modifying attention output projection weights (attn.o_proj) on a per-layer basis, while maintaining a KL divergence of just 0.0097 from the original model — preserving general behavior almost entirely.

cspenn/Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8 (this repo) — The heretic weights quantized for Apple Silicon using mlx-lm's convert with a custom quant_predicate. A sensitivity-informed mixed strategy keeps routing and boundary layers at 8-bit while compressing the bulk expert weights to 4-bit. See Quantization Layout below.

---

Model Details

---

Quantization Layout

Default: 4-bit affine, group_size=64. Explicit 8-bit overrides are applied to 132 tensors across six categories based on sensitivity analysis and layer-position anchoring.

Design rationale: MoE expert weights tolerate 4-bit compression well because only 8 of 256 experts activate per token, limiting quantization noise accumulation. Routing gates, embeddings, and the most sensitive linear-attention projection are kept at 8-bit to preserve routing quality and reduce perplexity degradation. The first and last transformer blocks are anchored at 8-bit as an additional safety margin. All tensors use group_size=64.

---

Usage

mlx-lm (Python)

from mlx_lm import load, generate

model, tokenizer = load("cspenn/Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8")

messages = [{"role": "user", "content": "Explain mixture-of-experts in one paragraph."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True)
print(response)

mlx-lm (CLI)

mlx_lm.generate \
  --model cspenn/Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8 \
  --prompt "Explain mixture-of-experts in one paragraph." \
  --max-tokens 512

LM Studio

LM Studio supports MLX safetensors natively on Apple Silicon. Either:

• From HuggingFace: Search for cspenn/Qwen3.6-35B-A3B-heretic-MLX-Mixed-4-8 in the Discover tab.
• Local folder: Use Load Model from Disk and point to the downloaded folder. The chat_template.jinja and tokenizer.json are included, so chat templating works out of the box.

---

Important Caveats

• Text-only. config.json retains Qwen3_5MoeForConditionalGeneration (a VLM architecture class) and the image_token_id field because altering these would break mlx-lm's model loader. However, the vision encoder weights were not included in the safetensors; any attempt to pass image inputs will raise an error. This is intentional.
• Heretic variant. This model is derived from an abliterated base. It is uncensored and will comply with requests the original Qwen model would refuse. Use responsibly.
• Memory. ~64 GB unified memory is recommended for comfortable inference. The model loads in ~20 GB of weight but requires additional memory for the KV cache at long context lengths.
• Thinking mode. Qwen3.6 supports an explicit <think> reasoning mode. The heretic abliteration was applied without disabling this capability; it should remain functional.

---

License

Apache 2.0 — same as the upstream Qwen model. See LICENSE.

This derivative is distributed under the same terms. Commercial use is permitted.

---

Credits

• Qwen team (Alibaba) — original Qwen3.6-35B-A3B architecture and weights.
• tvall43 — Heretic v1.2.0 abliteration producing the decensored base.
• mlx-lm — Apple's MLX framework and convert utility used for quantization.
• Quantization — produced by cspenn using a custom quant_predicate informed by OptiQ sensitivity analysis.