qwen3.6-27b-nvfp4-mm

Qwen3.6-27B-NVFP4-mm

NVFP4-A16 quantized build of a Qwen3-VL dense multimodal checkpoint, optimized for inference on Nvidia DGX Spark (GB10, Grace Blackwell, unified 128 GB) via vLLM. Sibling model to qwen3.6-35b-a3b-nvfp4-mm (MoE) — both can run side-by-side on a single GB10 thanks to unified memory.

• Quantization: NVFP4A16 (LLM Compressor QuantizationModifier, scheme NVFP4A16)
• Linear layers quantized; vision tower / merger / MoE gates / lm_head / linear_attn kept in original precision (see recipe.yaml)
• Weight file: model.safetensors (~26 GB)
• Format: compressed-tensors (loaded natively by vLLM >= 0.10 with NVFP4 GEMM backend)

Files

Hardware target

• Nvidia DGX Spark — GB10 Grace Blackwell, 128 GB unified CPU+GPU memory, ARM aarch64, CUDA 13.0
• Single GB10 node; tensor-parallel-size = 1
• ~26 GB weights + KV-cache; coexists with the 35B-A3B sibling on the same box

Install — vLLM environment (DGX Spark)

Building the vllm-nvfp4-env environment: see BUILD_ENV.md for the full procedure (PyTorch + vLLM from source for CUDA 13 / aarch64 / sm121, SDPA + NVFP4 shims, frozen requirements, smoke test). Shortcut: bash scripts/build-vllm-nvfp4-env.sh once the torch wheel has been built (step 1 in BUILD_ENV.md).

Note for 27B dense: this model requires the sdpa_fix shim on PYTHONPATH. Without it, vLLM crashes on the first attention forward pass on GB10 (the CUTLASS flash kernels are built for sm80-sm100 and produce NaN on sm121). The build script copies the shim to ~/llm/scripts/sdpa_fix/ automatically.

Once the environment is installed, download the weights for this model:

source ~/llm-test/vllm-nvfp4-env/bin/activate
hf auth login   # token with read permission
hf download <your-org-or-user>/qwen3.6-27b-nvfp4-mm \
  --local-dir ~/llm/models/Qwen3.6-27B-NVFP4-mm \
  --local-dir-use-symlinks False

Run — native vLLM (recommended)

source ~/llm-test/vllm-nvfp4-env/bin/activate
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_USE_FLASHINFER_MOE_FP4=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# SDPA fix shim — required for 27B dense on Blackwell
PYTHONPATH="$HOME/llm/scripts/sdpa_fix:${PYTHONPATH:-}" \
vllm serve ~/llm/models/Qwen3.6-27B-NVFP4-mm \
  --served-model-name qwen3.6-27b-nvfp4-mm \
  --host 0.0.0.0 --port 8014 \
  --tensor-parallel-size 1 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.55 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --trust-remote-code

Cold start: ~2-5 min. The server is ready when INFO: Application startup complete. appears in the log. Probe:

curl -s http://localhost:8014/v1/models | jq '.data[0].id'
# -> "qwen3.6-27b-nvfp4-mm"

When running alongside the 35B sibling: start 35B first on port 8015 with --gpu-memory-utilization 0.90, wait until it is ready, then start 27B on 8014 with 0.55. Order matters — 35B needs more free memory for bursts, while 27B dense fits in the remaining unified-memory buffer.

Sample request (OpenAI-compatible chat-completions)

curl -s http://localhost:8014/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b-nvfp4-mm",
    "messages": [
      {"role":"user","content":[
        {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,<...>"}},
        {"type":"text","text":"Extract invoice fields as JSON."}
      ]}
    ],
    "temperature": 0.0,
    "top_p": 0.8,
    "max_tokens": 2048
  }' | jq

The Qwen3-VL image processor rejects very high resolutions — downscale to ~1536 px on the longest edge before sending (300 DPI A4 is too large).

Notes & gotchas

• NVFP4 backend: VLLM_NVFP4_GEMM_BACKEND=marlin (Blackwell). Additionally for 27B dense: VLLM_TEST_FORCE_FP8_MARLIN=1, VLLM_USE_FLASHINFER_MOE_FP4=0.
• SDPA shim: without PYTHONPATH=~/llm/scripts/sdpa_fix, 27B dense crashes on GB10 on the first attention forward pass. The shim forces a stable SDPA backend.
• --gpu-memory-utilization 0.55: leaves ~45% of memory for the 35B sibling when both are running. Solo, this can be raised to 0.85.
• --max-model-len 32768: trimmed from 131k for a faster KV-cache.
• --enforce-eager deliberately omitted: 27B dense handles graph capture fine; 35B-A3B (MoE) uses --enforce-eager to avoid hangs.
• License: inherits Apache-2.0 from the base Qwen3-VL.

Provenance

Quantized from a Qwen3-VL 32B (dense, instruct) checkpoint with LLM Compressor — see recipe.yaml. Architecture: Qwen3-VL (vision encoder + dense LLM).

License

Apache-2.0 — same as the base Qwen3-VL model. See LICENSE for the full text and NOTICE for attribution.

Acknowledgements

This work stands on the shoulders of many open contributions. In particular:

• Qwen team @ Alibaba Cloud — base Qwen3-VL checkpoints
• sakamakismile — first public NVFP4 quantization of Qwen3.6-27B; their checkpoint exposed the swizzled-vs-linear scale-layout edge case that scripts/nvfp4_fix/ patches
• prithivMLmods — alternative NVFP4 quantization with ignore: linear_attn (avoiding the size_n=96 ∤ tile_n=64 Marlin-tile mismatch); their recipe choice is mirrored here
• Avarok — "We unlocked NVFP4 on DGX Spark" blog post + reference Docker image (avarok/dgx-vllm-nvfp4-kernel); the env-var combination that makes NVFP4 work on GB10 comes directly from their writeup
• Neural Magic / vLLM team — LLM Compressor (quantization tooling) and vLLM (inference engine)
• NVIDIA — CUDA, cuDNN, CUTLASS, the NVFP4 numerical format
• PyTorch, flashinfer, Hugging Face teams

Full attribution and links are in NOTICE. If you build on this repo, please keep that file intact and add your own acknowledgements alongside.