Qwen3.5-397B-A17B-NVFP4

Model Overview

• Model Architecture: Qwen3_5MoeForConditionalGeneration
  • Input: Text, Image, Video
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI300/MI350/MI355 (emulation)
• ROCm: 7.2.2
• PyTorch: 2.10.0
• Transformers: 5.14.0
• Operating System(s): Linux
• Inference Engine: vLLM
• Model Optimizer: AMD-Quark (v0.12)
  • Quantized layers: Experts in language model only
  • Weight quantization: MOE-only, NVFP4, Static
  • Activation quantization: MOE-only, NVFP4, Dynamic

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. The weights and activations are quantized to NVFP4.

Quantization scripts:

exclude_layers="lm_head model.visual.* mtp.* *mlp.gate *shared_expert* *shared_expert_gate* *linear_attn.* *self_attn.*"
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export MODEL_DIR=Qwen/Qwen3.5-397B-A17B
export output_dir=amd/Qwen3.5-397B-A17B-NVFP4
python3 quantize_quark.py \
--model_dir $MODEL_DIR \
--quant_scheme nvfp4\
--num_calib_data 128 \
--multi_gpu balanced \
--exclude_layers $exclude_layers \
--model_export hf_format  \
--output_dir $output_dir

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend.

Evaluation

The model was evaluated on GSM8K benchmarks.

Accuracy

<table> <tr> <td><strong>Benchmark</strong> </td> <td><strong>Qwen/Qwen3.5-397B-A17B </strong> </td> <td><strong>amd/Qwen3.5-397B-A17B-NVFP4(this model)</strong> </td> <td><strong>Recovery</strong> </td> </tr> <tr> <td>gsm8k (flexible-extract) </td> <td>95.38 </td> <td>94.47 </td> <td>99.04% </td> </tr> </table>

Reproduction

The GSM8K result was obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260603.

Install the lm-eval (Version: 0.4.12) in container first.

pip install lm-eval[api]

Evaluating model in a new terminal

export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
lm_eval   --model vllm   
          --model_args pretrained=amd/Qwen3.5-397B-A17B-NVFP4,tensor_parallel_size=8,max_model_len=262144,gpu_memory_utilization=0.90,max_gen_toks=2048,trust_remote_code=True,reasoning_parser=qwen3 \
          --tasks gsm8k \
          --num_fewshot 5 \
          --batch_size auto

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.