Qwen3.6-35B-A3B-INT8-W8A8

Qwen3.6-35B-A3B-INT8-W8A8

INT8 (W8A8) quantization of Qwen/Qwen3.6-35B-A3B — a hybrid Mixture-of-Experts model (256 experts, top-8, ~3B active) with GatedDeltaNet linear-attention, a vision tower and an MTP head.

Quantization

• Symmetric INT8 weights (per-channel, MSE observer) + INT8 dynamic per-token activations (llm-compressor). int8 is robust, so no AWQ/SmoothQuant smoothing is needed.
• Quantized: the MoE expert FFNs only (the bulk of the weights).
• Kept bf16 (quality-sensitive): self_attn, the router (mlp.gate), shared_expert, GatedDeltaNet (linear_attn), lm_head, embeddings, vision tower, MTP head.
• Format: compressed-tensors (int-quantized). Full recipe: recipe.yaml.

INT8 W8A8 is near-lossless; for the smallest footprint use the INT4-W4A16 variant (GSM8K 96.8% / MMLU-Pro 80.2%).

Usage (vLLM)

vllm serve Avesed/Qwen3.6-35B-A3B-INT8-W8A8 \
  --tensor-parallel-size 2 --trust-remote-code --reasoning-parser qwen3

Served via vLLM's INT8 MoE path (works on Ampere sm_80 / sm_86).

Quantized with vllm-ampere-optimized/quantize.