Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2

Model Overview

• Model Architecture: Qwen3_5MoeForConditionalGeneration
  • Input: Text, Image, Video
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350 / MI355
• ROCm: 7.2.0
• PyTorch: 2.9.1
• Transformers: 5.3.0
• Operating System(s): Linux
• Inference Engine: SGLang
• Model Optimizer: AMD-Quark (v0.12)
  • Quantized layers: All MoE experts in the language model, including the shared expert (the shared expert is also fused into the MoE kernel for faster decode).
  • Weight quantization:
    • self_attn and linear_attn Perchannel, FP8E4M3, Static
    • MOE OCP MXFP4, Static
  • Activation quantization:
    • self_attn and linear_attn Pertoken, FP8E4M3, Dynamic
    • MOE OCP MXFP4, Dynamic

This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

MODEL_PATH="Qwen/Qwen3.5-397B-A17B"
OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2"

cd /path/to/Quark

python3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir "${MODEL_PATH}" \
  --quant_scheme mxfp4 \
  --layer_quant_scheme '*linear_attn*' ptpc_fp8 \
  --layer_quant_scheme '*self_attn*' ptpc_fp8 \
  --no_trust_remote_code \
  --model_export hf_format \
  --skip_evaluation \
  --multi_gpu balanced \
  --exclude_layers "lm_head" "*mlp.gate" "mtp.*" "model.visual.*" "*shared_expert_gate*" \
  --output_dir "${OUTPUT_DIR}"

For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.

Evaluation

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B checkpoint, evaluated with the identical recipe on SGLang.

Accuracy

GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.

Reproduction

The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.

1. Serve the model with SGLang:

export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export SGLANG_USE_AITER_FP8_PER_TOKEN=1

python3 -m sglang.launch_server \
    --model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --tensor-parallel-size 2 \
    --attention-backend aiter \
    --mem-fraction-static 0.8 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --watchdog-timeout 1200 \
    --disable-radix-cache \
    --reasoning-parser qwen3 \
    --page-size 16 \
    --enable-aiter-allreduce-fusion

1. Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy). This uses InferenceX's gsm8k.yaml, whose only change vs. the stock lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:

lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k.yaml \
  --num_fewshot 5 \
  --model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \
  --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.