qwen3.6-35b-a3b-nvfp4-mm

Qwen3.6-35B-A3B-NVFP4-mm

NVFP4-A16 quantized build of a Qwen3-VL multimodal MoE checkpoint, optimized for inference on Nvidia DGX Spark (GB10, Grace Blackwell, unified 128 GB) via vLLM.

• Quantization: NVFP4A16 (LLM Compressor QuantizationModifier, scheme NVFP4A16)
• Linear layers quantized; vision tower / merger / MoE gates / lm_head / linear_attn / MTP layers kept in original precision (see recipe.yaml)
• Weight file: model.safetensors (~22 GB)
• Format: compressed-tensors (loaded natively by vLLM >= 0.10 with NVFP4 GEMM backend)

Files

Hardware target

• Nvidia DGX Spark — GB10 Grace Blackwell, 128 GB unified CPU+GPU memory, ARM aarch64, CUDA 13.0
• Works on a single GB10 node; tensor-parallel-size = 1
• Coexists with a 27B sibling model on the same box (~50 GB total when both are up)

Install — vLLM environment (DGX Spark)

Building the vllm-nvfp4-env environment: see BUILD_ENV.md for the full procedure (PyTorch + vLLM from source for CUDA 13 / aarch64 / sm121, SDPA + NVFP4 shims, frozen requirements, smoke test). Shortcut: bash scripts/build-vllm-nvfp4-env.sh once the torch wheel has been built (step 1 in BUILD_ENV.md).

Once the environment is installed, download the weights for this model:

source ~/llm-test/vllm-nvfp4-env/bin/activate
hf auth login   # token with read permission
hf download <your-org-or-user>/qwen3.6-35b-a3b-nvfp4-mm \
  --local-dir ~/llm/models/Qwen3.6-35B-A3B-NVFP4-mm \
  --local-dir-use-symlinks False

Run — native vLLM (recommended for DGX Spark)

source ~/llm-test/vllm-nvfp4-env/bin/activate
export VLLM_NVFP4_GEMM_BACKEND=marlin
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve ~/llm/models/Qwen3.6-35B-A3B-NVFP4-mm \
  --served-model-name qwen3.6-35b-a3b-nvfp4-mm \
  --host 0.0.0.0 --port 8015 \
  --tensor-parallel-size 1 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --trust-remote-code \
  --enforce-eager

Cold start: ~2-5 min. Ready when:

INFO: Application startup complete.

Probe:

curl -s http://localhost:8015/v1/models | jq '.data[0].id'
# -> "qwen3.6-35b-a3b-nvfp4-mm"

Run — Docker (alternative)

docker run -d --name vllm-qwen36-35b-a3b-nvfp4-mm \
  --gpus all --ipc=host \
  -p 8015:8000 \
  -v ~/llm/models:/models:ro \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --entrypoint vllm \
  vllm/vllm-openai:gemma4-cu130 \
  serve /models/Qwen3.6-35B-A3B-NVFP4-mm \
    --served-model-name qwen3.6-35b-a3b-nvfp4-mm \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 1 \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --max-num-seqs 64 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --trust-remote-code \
    --enforce-eager

Sample request (OpenAI-compatible chat-completions)

curl -s http://localhost:8015/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-35b-a3b-nvfp4-mm",
    "messages": [
      {"role":"user","content":[
        {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,<...>"}},
        {"type":"text","text":"Extract invoice fields as JSON."}
      ]}
    ],
    "temperature": 0.0,
    "top_p": 0.8,
    "max_tokens": 2048
  }' | jq

For Qwen3-VL the image processor rejects very high-res inputs; downscale to ~1536 px on the longest edge before sending (300 DPI A4 is too large).

Notes & gotchas

• NVFP4 backend: set VLLM_NVFP4_GEMM_BACKEND=marlin (Blackwell). Without it vLLM falls back to a slow path.
• --enforce-eager: graph capture has occasionally hung on this build — eager mode is the safe default.
• --max-model-len 32768: trimmed from the model's full 131k context for faster KV-cache; raise it if your prompts need more.
• Coexistence with 27B sibling (Qwen3.6-27B-NVFP4-mm, port 8014): both fit thanks to GB10 unified memory. Recommended GPU utilization split: 27B = 0.55, 35B = 0.90 (35B started first, 27B fills the remainder).
• License: inherits Apache-2.0 from base Qwen3-VL.

Provenance

Quantized from a Qwen3-VL 30B-A3B (MoE, 35B total params) instruct checkpoint with LLM Compressor — see recipe.yaml. Original architecture: Qwen3-VL (vision encoder + LLM with sparse-MoE feed-forward).

License

Apache-2.0 — same as the base Qwen3-VL model. See LICENSE for the full text and NOTICE for attribution.

Acknowledgements

This work stands on the shoulders of many open contributions. In particular:

• Qwen team @ Alibaba Cloud — base Qwen3-VL checkpoints
• sakamakismile — first public NVFP4 quantization of Qwen3.6-27B; their checkpoint exposed the swizzled-vs-linear scale-layout edge case that scripts/nvfp4_fix/ patches
• prithivMLmods — alternative NVFP4 quantization that demonstrated the ignore: linear_attn recipe choice mirrored here
• Avarok — "We unlocked NVFP4 on DGX Spark" blog post + reference Docker image (avarok/dgx-vllm-nvfp4-kernel); the env-var combination that makes NVFP4 work on GB10 comes directly from their writeup
• Neural Magic / vLLM team — LLM Compressor (quantization tooling) and vLLM (inference engine)
• NVIDIA — CUDA, cuDNN, CUTLASS, the NVFP4 numerical format
• PyTorch, flashinfer, Hugging Face teams

Full attribution and links are in NOTICE. If you build on this repo, please keep that file intact and add your own acknowledgements alongside.