Qwen3-VL-8B-Thinking-NVFP4

Qwen3-VL-8B-Thinking NVFP4A16 — vLLM / RTX 5060 Ti Test

This repository contains a locally quantized Qwen3-VL-8B-Thinking model using NVFP4A16 / compressed-tensors, tested with vLLM nightly on a consumer NVIDIA Blackwell GPU.

This model was prepared for vLLM inference with multimodal support enabled. The language backbone was quantized to NVFP4A16, while the visual encoder was kept in BF16 to avoid Marlin tile-size incompatibilities in Qwen3-VL visual MLP layers.

This model card section documents the local runtime test performed by Murilo Vieira on RTX 5060 Ti 16GB.

Quantization Summary

Why the Visual Encoder Was Not Quantized

During the initial NVFP4 conversion, vLLM failed when preparing some Qwen3-VL visual MLP layers for Marlin FP4 execution. The problematic layers had output dimensions such as:

model.visual.blocks.*.mlp.linear_fc1 | out_features=4304

Marlin FP4 packing requires compatible tile dimensions, and 4304 is not divisible by 64. To make the model load successfully in vLLM, the visual encoder was excluded from NVFP4 quantization.

The final working layout is:

Language backbone: NVFP4A16
Visual encoder: BF16
lm_head: BF16
Runtime: vLLM compressed-tensors

Tested Hardware

vLLM Runtime Configuration

model: /models/Qwen3-VL-8B-Thinking-NVFP4
dtype: auto
gpu_memory_utilization: 0.93
max_model_len: 32768
max_num_batched_tokens: 8192
max_num_seqs: 4

enable_auto_tool_choice: true
tool_call_parser: qwen3_xml
reasoning_parser: qwen3

enable_chunked_prefill: true
performance_mode: interactivity
enforce_eager: false
trust_remote_code: true
tensor_parallel_size: 1

port: 8000
served_model_name:
  - Qwen3-VL

structured_outputs_config:
  backend: xgrammar
  disable_any_whitespace: true
  enable_in_reasoning: false

enable_server_load_tracking: true
kv_cache_metrics: true
kv_cache_metrics_sample: 0.05

limit_mm_per_prompt:
  image: 3
  video: 1
  audio: 1

mm_processor_kwargs:
  min_pixels: 3136
  max_pixels: 10035200
  fps: 1.0

media_io_kwargs:
  video:
    num_frames: 32
    fps: 2

data_parallel_size: 1
video_pruning_rate: 0.75

mm_encoder_tp_mode: data
mm_processor_cache_type: shm

kv_cache_dtype: auto
enable_prefix_caching: true
prefix_caching_hash_algo: xxhash

async_scheduling: true

compilation_config:
  pass_config:
    fuse_allreduce_rms: true
    fuse_attn_quant: true
    eliminate_noops: true

generation_config: vllm
skip_mm_profiling: true

Startup and Memory Statistics

Throughput Observed

Runtime Behavior

• vLLM successfully resolved the architecture as Qwen3VLForConditionalGeneration.
• vLLM detected the model as quantization=compressed-tensors.
• FlashAttention was used for the main attention backend.
• FlashAttention was also used for ViT and multimodal encoder attention.
• FlashInfer was used for top-k / top-p sampling.
• torch.compile was enabled and cached the graph for compile range (1, 8192).
• CUDA graph capture completed successfully for decode.
• Multimodal warmup completed successfully.
• The OpenAI-compatible API server started successfully on port 8000.

Docker Compose Used

vllm-vl:
  image: vllm/vllm-openai:nightly-x86_64
  container_name: vllm-vl
  hostname: vllm-vl
  restart: unless-stopped
  runtime: nvidia
  network_mode: host
  ipc: host
  shm_size: '16gb'
  volumes:
    - /mnt/dados/storage/cache/huggingface:/root/.cache/huggingface
    - /mnt/dados/storage/cache/vllm-cache:/root/.cache/vllm
    - /mnt/dados/storage/models/Qwen:/models:ro
    - /mnt/dados/storage/config:/configs:ro
  environment:
    - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
    - TORCH_CUDA_ARCH_LIST=12.0
    - CUDA_DEVICE_ORDER=PCI_BUS_ID
    - NVIDIA_VISIBLE_DEVICES=all
    - NVIDIA_DRIVER_CAPABILITIES=all
    - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
    - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    - OMP_NUM_THREADS=14
    - VLLM_NO_USAGE_STATS=1
    - DO_NOT_TRACK=1
    - VLLM_TEST_FORCE_FP8_MARLIN=1
    - VLLM_MARLIN_USE_ATOMIC_ADD=1
    - TORCH_MATMUL_PRECISION=high
    - NVIDIA_FORWARD_COMPAT=1
    - NVIDIA_DISABLE_REQUIRE=1
    - VLLM_USE_FLASHINFER_SAMPLER=1
    - VLLM_NVFP4_GEMM_BACKEND=marlin
    - VLLM_VIDEO_LOADER_BACKEND=opencv
    - VLLM_TARGET_DEVICE=cuda
    - VLLM_FLOAT32_MATMUL_PRECISION=high
    - VLLM_USE_STANDALONE_COMPILE=1
    - VLLM_ENABLE_V1_MULTIPROCESSING=1
    - VLLM_V1_OUTPUT_PROC_CHUNK_SIZE=128
    - VLLM_SKIP_MODEL_NAME_VALIDATION=1
    - VLLM_LOGGING_LEVEL=INFO
    - VLLM_LOG_STATS_INTERVAL=10
    - VLLM_XGRAMMAR_CACHE_MB=1024
    - VLLM_IMAGE_FETCH_TIMEOUT=10
    - VLLM_AUDIO_FETCH_TIMEOUT=30
    - VLLM_MEDIA_CACHE=/root/.cache/vllm/media
    - VLLM_MEDIA_CACHE_MAX_SIZE_MB=10240
    - VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=100
  command: >
    --config /configs/Qwen3-VL.yaml
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['1']
            capabilities: [gpu]

Suggested vLLM Command

vllm serve /models/Qwen3-VL-8B-Thinking-NVFP4 \
  --trust-remote-code \
  --served-model-name Qwen3-VL \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.93 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 4 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --prefix-caching-hash-algo xxhash \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --performance-mode interactivity \
  --port 8000

Known Runtime Warnings

The following warnings were observed and are not necessarily fatal:

• VLLM_BUILD_COMMIT, VLLM_BUILD_PIPELINE, VLLM_BUILD_URL, and VLLM_IMAGE_TAG were detected as unknown vLLM environment variables.
• Qwen2VLImageProcessorFast deprecation warnings were emitted by Transformers.
• fuse_attn_quant was reported as incompatible with piecewise CUDA Graphs when graph partitioning was disabled.
• AllReduce fusion was disabled because tp_size <= 1.
• MLA attention + quant fusion was enabled, but no MLA layers were found.
• Triton JIT compilation occurred during early inference for _compute_slot_mapping_kernel, _bilinear_pos_embed_kernel, and rotary_kernel.

Reproducibility Notes

These results are from a single local test run and should be treated as environment-specific. Throughput may vary depending on:

• prompt length,
• output length,
• number and resolution of images,
• video frame count,
• sampling settings,
• driver/CUDA versions,
• vLLM nightly build,
• whether torch.compile cache is already warm,
• FlashAttention / FlashInfer backend behavior,
• KV cache dtype,
• context length,
• multimodal preprocessing settings.

Status

✅ Loaded successfully
✅ OpenAI-compatible server started
✅ Multimodal warmup completed
✅ Qwen3 XML tool parser enabled
✅ Long context configured at 32,768 tokens
✅ NVFP4A16 compressed-tensors runtime working
✅ Tested on RTX 5060 Ti 16GB