Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM

Qwen3.5-35B-A3B — Dynamic 2/3-bit MLX VLM (11 GB, Vision + Text on M4 Mini 16GB)

Part of the Qwen3.5-35B-A3B · Alis builds collection.

✅ This is the VLM (Vision-Language) version with vision encoder included in bf16. For text-only with more KV headroom: avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw

Extreme dynamic quantization to run a 35B multimodal MoE model on 16GB Apple Silicon. Expert 2-bit + Attention/GDN 3-bit + Router bf16 + Vision encoder bf16. Converted with mlx_vlm, which includes the full vision encoder for image understanding.

---

🚨 CRITICAL: 16GB Mac Users — Read This First

VLM is even tighter on 16GB than the text-only version (12.5 GB peak vs 11.3 GB). Follow ALL steps or the model WILL crash:

1. Thinking Mode MUST be OFF

python -m mlx_vlm generate \
    --model avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM \
    --prompt 'Describe this image.' \
    --image photo.jpg \
    --max-tokens 200

Always set --max-tokens (default is unlimited → OOM crash).

2. Close ALL Other Apps

16.0 GB total
- 3.5 GB  macOS + background services
- 12.5 GB model (peak, with vision encoder)
= 0.0 GB  ← ZERO headroom!

Close absolutely everything — Safari, Chrome, Slack, Discord, KakaoTalk. Even one tab can cause OOM.

3. Sampling Parameters

4. Strongly Recommended: Headless + SSH

VLM on 16GB is at the absolute limit. Headless mode frees ~0.5 GB from WindowServer.

Quick Checklist for 16GB VLM

💡 24GB+ users: VLM runs comfortably with thinking ON and long prompts.

💡 16GB text-only users: Consider the text-only version which has 1.2 GB KV headroom instead of near-zero.

---

Key Specs

---

Purpose

This model was built to run a 35B-class MoE language model on an Apple M4 Mac Mini with only 16GB unified memory.

Limitations of existing quantized models:

• Uniform 4-bit (~21GB): Doesn't fit in 16GB
• Uniform 3-bit (~14GB, peak 15.3GB): Barely fits but no room for KV cache
• Uniform 2-bit (~11GB): Complete quality collapse (gibberish output)

This model solves the problem through role-based dynamic quantization:

• Core pathways active every token (Attention, GatedDeltaNet) → protected at 3-bit
• Experts where only 8 of 256 are active per token → aggressively compressed to 2-bit
• Router and Norms → kept at bf16, never quantized

Result: 11GB multimodal model with vision + 100% Korean quality + English/coding/reasoning.

---

Quantization Strategy

Per-Layer Bit Allocation

Why 2-bit Works for Experts Only

This exploits a key property of MoE architecture:

• Only 8 of 256 experts activate per token (3.1% activation rate)
• Quantization error affects only 3.1% of pathways
• As long as the Router (bf16) correctly selects experts, slight errors in selected experts are tolerable
• In dense models, all parameters are active every token → 2-bit errors accumulate → collapse

Sensitivity Analysis Validation

Per-layer relative error measured on 8-domain calibration data (Korean, English, code, reasoning):

Expert layers are 6.5% more robust than attention layers, confirming the MoQE paper (Kim et al., 2023). Edge layers (L0-2, L37-39) show no additional sensitivity (edge/mid ratio = 1.00x), so APEX-style edge protection is unnecessary for this model.

Failed Approaches (Lessons Learned)

---

Usage

Installation

pip install mlx-vlm
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

Image + Text Generation

python -m mlx_vlm generate \
    --model avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM \
    --prompt 'Describe this image in detail.' \
    --image /path/to/image.jpg \
    --max-tokens 200

Text-Only Generation

python -m mlx_vlm generate \
    --model avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM \
    --prompt 'What is the capital of South Korea?' \
    --max-tokens 200

Python API

from mlx_vlm import load, generate

model, processor = load("avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM")

# Image understanding
response = generate(
    model, processor,
    prompt="이 이미지를 설명해주세요.",
    image="/path/to/image.jpg",
    max_tokens=500,
)
print(response)

API Server (OpenAI-Compatible)

python -m mlx_vlm.server \
    --model avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM \
    --port 8888

---

Recommended Settings (Based on Qwen Official)

Thinking OFF (Agents, General Chat, API Serving)

Thinking ON (Math, Coding, Complex Reasoning)

Special Notes for This 2.58 BPW Quantized Model

⚠️ This model uses extreme 2.58 BPW quantization. Repetition loops are slightly more likely than the original. Follow these recommendations:

• Always set presence_penalty to 1.5 or higher. Setting it to 0 may cause repetition loops.
• Never set temperature to 0. Greedy decoding causes quality degradation and repetitions in quantized models.
• For long outputs, set max_tokens sufficiently high (default 200 is too short).

Server Launch Example

python -m mlx_vlm.server \
    --model avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM \
    --port 8888 \
    --chat-template-args '{"enable_thinking": false}' \
    --temp 0.7 --top-p 0.8 --top-k 20

API Call Examples

# Thinking OFF (Agent/General Chat)
curl http://localhost:8888/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM",
        "messages": [{"role": "user", "content": "Tell me about Korean traditional foods."}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "presence_penalty": 1.5,
        "extra_body": {"top_k": 20},
        "chat_template_kwargs": {"enable_thinking": false}
    }'

# Thinking ON (Math/Reasoning)
curl http://localhost:8888/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM",
        "messages": [{"role": "user", "content": "Calculate 12345 × 6789."}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "presence_penalty": 1.5,
        "extra_body": {"top_k": 20},
        "chat_template_kwargs": {"enable_thinking": true}
    }'

---

⚠️ Important Notes

Memory Budget (M4 Mini 16GB)

Total Memory:        16.0 GB
- macOS Overhead:    -3.5 GB
- Model (peak):      -12.5 GB
= KV Cache Headroom:  0.5 GB (tight but functional)

KV Cache and Context Length

Qwen3.5-35B-A3B's hybrid architecture makes KV cache extremely efficient:

• Full attention: Only 10 of 40 layers (remaining 30 are GatedDeltaNet)
• GQA KV heads: Only 2 (extremely low)
• 4-bit KV per token: ~5 KB
• GatedDeltaNet state: ~33 MB (fixed, independent of context length)

Note: VLM version has tighter KV headroom (~0.5 GB) than the text-only version (~1.2 GB). For long-context tasks beyond 100K tokens, use the text-only version instead.

Thinking Mode

• Default: Thinking ON (outputs internal reasoning process)
• For agents/API: Must set enable_thinking=False
• No quality degradation with Thinking OFF (Korean quality remains 100%)
• Thinking ON improves accuracy on complex math/reasoning problems

When NOT to Use This Model

• Long context (128K+): Use the text-only version which has 1.2 GB KV headroom
• 24GB+ Macs: Uniform 3-bit or 4-bit models provide better quality
• GPU servers: GPTQ/AWQ quantization is more suitable

Text-Only vs VLM Version Comparison

Which Version Should You Use?

• Choose Text-Only if you need maximum context length (245K+), fastest speed, or are building text-only agents/tools
• Choose VLM if you need image understanding, document analysis, or any visual input processing
• Both versions use the exact same language model weights — text quality is identical
• The only difference is the 0.83 GB bf16 vision encoder, which reduces KV headroom on 16GB devices

---

Benchmarks

Korean Quality Test (M4 Mac Mini 16GB)

20 Korean prompts × Thinking OFF, max_tokens=200:

Test prompts covered: capital city, traditional foods, kimchi recipe, Korean history, Seoul tourism, Hangul origins, seasonal weather, bulgogi recipe, economic industries, traditional medicine, education system, Jeju Island, IT industry, traditional music, holidays, bibimbap, healthcare system, Korean grammar, traditional architecture, K-pop global success.

Inference Speed

Quantization Profile Comparison (Same Hardware)

lm-eval Benchmarks (0-shot, text-only — same language model weights)

2.9pp average loss for 29% size reduction — and 3-bit doesn't fit in 16GB, so v3 is the only working option.

---

Reproducing the Quantization

Uses mlx-vlm's custom quant_predicate API:

from mlx_vlm.convert import convert
import re

def qwen35_v3(layer_path, layer):
    # Router: bf16 (never quantize)
    if layer_path.endswith("mlp.gate"):
        return False
    if "shared_expert_gate" in layer_path:
        return False
    # Norms: bf16
    if "norm" in layer_path and "proj" not in layer_path:
        return False
    # GDN parameters: bf16
    if any(x in layer_path for x in ["dt_bias", "A_log", "conv1d"]):
        return False
    # Embed/lm_head: 3-bit
    if "embed_tokens" in layer_path or "lm_head" in layer_path:
        return {"bits": 3, "group_size": 64}
    # Shared expert: 3-bit
    if "shared_expert" in layer_path:
        return {"bits": 3, "group_size": 64}
    # Full attention: 3-bit
    if "self_attn" in layer_path:
        return {"bits": 3, "group_size": 64}
    # Linear attention (GatedDeltaNet): 3-bit
    if "linear_attn" in layer_path:
        return {"bits": 3, "group_size": 64}
    # Routed experts: 2-bit (primary compression target)
    if "switch_mlp" in layer_path:
        return {"bits": 2, "group_size": 64}
    # Vision encoder: bf16 (never quantize)
    if "visual" in layer_path or "vision" in layer_path:
        return False
    # Everything else: 2-bit
    if hasattr(layer, "to_quantized"):
        return {"bits": 2, "group_size": 64}
    return False

convert(
    hf_path="Qwen/Qwen3.5-35B-A3B",
    mlx_path="./qwen35-dynamic-v3-vlm",
    quantize=True,
    quant_predicate=qwen35_v3,
)
# [INFO] Quantized model with 2.749 bits per weight.

---

Project Background

This model is the successor to the Gemma 4 26B MoE extreme quantization project.

The Gemma4 project achieved 11GB at 92% Korean quality with OptiQ B++++ strategy, but failed to deploy on M4 Mac Mini due to an MLX bug where the 128-expert gather_mm Metal kernel malfunctions on M4 base (10 GPU cores).

Reasons for switching to Qwen3.5:

1. M4 Compatibility: 256 experts but different MLX implementation — works on M4 base
2. KV Cache Efficiency: GQA 2 heads + GatedDeltaNet hybrid → 24x more efficient than Gemma4
3. Korean Quality: 201 language support, 100% OK without QLoRA

Gemma4 vs Qwen3.5 Comparison

---

Build Environment

• Quantization: M3 Ultra Mac Studio 512GB
• Deployment/Validation: M4 Mac Mini 16GB (macOS 26.3.1)
• Framework: MLX 0.31.1, mlx-vlm
• Date: April 11, 2026
• Author: @avlp12 + Claude Opus

License

This model inherits the Apache License 2.0 from the original Qwen3.5-35B-A3B.

Citation

@misc{qwen35-dynamic-v3-2026,
  title   = {Qwen3.5-35B-A3B Dynamic v3: 2.58 BPW Mixed-Precision for 16GB Apple Silicon},
  author  = {avlp12},
  year    = {2026},
  url     = {https://huggingface.co/avlp12/Qwen3.5-35B-A3B-Alis-MLX-Dynamic-2.6bpw-VLM},
  note    = {Expert 2-bit + Attention/GDN 3-bit + Router bf16 dynamic quantization}
}

Acknowledgments

• Qwen Team — Qwen3.5 model and Apache 2.0 license
• Apple MLX Team — MLX framework and quant_predicate API
• APEX-Quant — Inspiration for layer-wise precision gradient strategy
• Unsloth — Dynamic 2.0 quantization benchmarks and GGUF reference data