Qwen3.6-35B-A3B-MXFP4-MOE-RTN-ROUTERDISTILL

Qwen3.6-35B-A3B-MXFP4-MOE-RTN-ROUTERDISTILL

Qwen/Qwen3.6-35B-A3B quantized with AMD Quark.

What is quantized

Quark's built-in qwen3_5_moe recipe quantizes the routed MoE experts only — 30,720 linears (40 layers x 256 experts x 3 projections). 462 modules are excluded and remain BF16:

lm_head, model.visual.* (vision tower), mtp.*, *mlp.gate, *shared_expert_gate*, *.linear_attn.*, *.self_attn.*, *.shared_expert.*

Because attention, the shared experts, the router and the vision tower all stay BF16, the size reduction is smaller than a whole-model quantization would give.

Reproduction

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir Qwen/Qwen3.6-35B-A3B \
  --device cuda --data_type bfloat16 --model_attn_implementation eager \
  --dataset pileval --num_calib_data 128 --seq_len 512 --batch_size 1 \
  --quant_scheme mxfp4 \
  --preserve_fused_experts \
  --model_export hf_format --export_weight_format real_quantized \
  --output_dir <out> --skip_evaluation

Notes

• Quantized on a single AMD Instinct MI355X (gfx950) with Quark 0.13+50dac6e159a.
• No calibration data influences this checkpoint: weights are RTN-rounded and activation scales are computed dynamically at runtime.
• The vision tower is not quantized, so this remains a full vision-language model.
• GGUF export is not possible for this architecture: Quark's GGUF writer maps only MODEL_ARCH.LLAMA and supports asymmetric INT4 per-group-32 weight-only on Llama2/Llama3, while this model is Qwen3_5MoeForConditionalGeneration. Safetensors is published instead.
• No accuracy evaluation has been run on this checkpoint. Benchmark before use.
• Experimental: additionally applies router distillation from the BF16 teacher, applied across layers. This is a research variant from an internal MoE quantization study, not a standard Quark recipe.

Evaluation

GSM8K (5-shot, max_gen_toks=2048) and WikiText-2 perplexity, measured with lm-evaluation-harness against the bf16 baseline (Qwen/Qwen3.6-35B-A3B).

Caveat on baseline methodology: the bf16 baselines above were originally distorted by two independent bugs we found and fixed before finalizing these numbers: (1) lm-evaluation-harness's default max_gen_toks=256 truncated this thinking model's <think> traces before reaching an answer (fixed: raised to 2048); (2) a real, reproducible bug in vLLM's Triton implementation of the chunk_gated_delta_rule custom op on this ROCm build causes some bf16 generations to collapse into a fixed degenerate phrase and stop early -- confirmed via a working reference PyTorch implementation (transformers eager, no custom kernel) producing coherent output on the identical prompt/weights, and confirmed in both eager and CUDA-graph-compiled vLLM modes (ruling out graph capture as the cause). Workaround: compilation_config={"custom_ops": ["-chunk_gated_delta_rule"]} forces the correct fallback path. Cross-validated against a 150-example CPU ground-truth run for both base models -- the bug's net effect on the aggregate GSM8K score was small, so the baseline numbers above (measured with the workaround applied) should be reliable. Separately, several checkpoints in this family genuinely score well above the bf16 baseline on GSM8K (a 15-45 point jump in some cases). We verified this is real, not a measurement artifact: it persists identically with and without the kernel-bug workaround applied. It most likely reflects quantization noise suppressing this reasoning model's tendency to enter long, sometimes unproductive chains of self-questioning on the unquantized baseline -- not a genuine capability improvement from quantization. WikiText-2 perplexity (likelihood-based, not generation-based, and unaffected by either bug) shows the expected small regression from quantization and is the more trustworthy signal of quantization quality here.