qwen3.6-35b-a3b-modelopt-nvfp4

Qwen3.6-35B-A3B-ModelOpt-NVFP4

This repository contains an optimized NVFP4 (Group Size 16) quantized checkpoint of Qwen3.6-35B-A3B-DSV4Pro-FP4 (a 35B parameter sparse Mixture-of-Experts model) produced using NVIDIA ModelOpt and formatted for native vLLM serving.

Model Summary

• Base Architecture: Qwen 3.5 MoE (Qwen3_5MoeForCausalLM)
• Quantization Method: NVIDIA ModelOpt NVFP4 (W4A16 per-group quantization)
• Group Size: GS16 (Required configuration for vLLM Marlin FP4 CUDA kernel execution)
• Total Model Size: ~19.66 GB safetensors weight shards (6 shards)
• Vocabulary Size: 248,320 tokens (intact)
• Vision Encoder & MTP: Vision encoder intact (visual.*), vocabulary intact, MTP heads stripped (mtp.*)

Group Size & vLLM Serving Note

Critical Serving Note: While Group Size 128 (GS128) shrinks footprint further, the vLLM Marlin FP4 CUDA kernel (marlin_mm) only supports a group size of 16. Attempts to serve GS128 will result in a engine crash (Invalid thread config). Therefore, GS16 is the only viable serving configuration.

Measured Benchmark Results

• GSM8K Math Reasoning (20 Representative Samples, temp=0.6, top_p=0.95): Evaluated under local test suite
• ARC-Challenge Science Reasoning (20 Representative Samples, temp=0.6, top_p=0.95): Evaluated under local test suite

Serving with vLLM

vllm serve Cadododoom/qwen3.6-35b-a3b-modelopt-nvfp4 \
  --served-model-name qwen3.6-35b-a3b-modelopt-nvfp4 \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --moe-backend marlin \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --max-model-len 112000 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.96 \
  --enable-prefix-caching \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000