Qwen-AgentWorld-35B-A3B-NVFP4-MTP

Qwen-AgentWorld-35B-A3B-NVFP4 with MTP Speculative Decoding

This is a modified version of the Qwen-AgentWorld-35B-A3B-NVFP4 model, enhanced with Multi-Token Prediction (MTP) weights to enable speculative decoding for approximately 1.7–1.9× decode speedup.

📝 Background & What Was Done

The AgentWorld NVFP4 fine-tune lost its MTP weights during training — they were never saved to the checkpoint. The base Qwen/Qwen3.6-35B-A3B model architecture supports MTP, so the weights can be borrowed from any compatible Qwen3.6-35B NVFP4 model that ships with them.

Additionally, the AgentWorld config.json was missing the re:^mtp.* entry in quantization_config.ignore, which caused vLLM's compressed-tensors handler to apply NVFP4 quantization to the MTP head. This made the internal parameter names mismatch the BF16 disk tensors, crashing with:

KeyError: 'layers.0.mlp.experts.routed_experts.w2_weight'

The working RedHatAI/Qwen3.6-35B-A3B-NVFP4 model has both the MTP tensors and the ignore entry, which is why it works out of the box.

How to Pull the Original Donor Model

To obtain the MTP head tensors, we use the nvidia/Qwen3.6-35B-A3B-NVFP4 model, which ships with its MTP head in BF16 (unquantized) inside the main checkpoint. Download it so vLLM can cache it locally:

HF_HOME=<vllm-models-dir> \
  hf download nvidia/Qwen3.6-35B-A3B-NVFP4

Modifications Made:

1. Merged MTP Tensors: The script scripts/merge_mtp_into_agentworld.py backs up the original AgentWorld snapshot to <hash>.orig_backup/ before touching anything. It extracts all 19 mtp.* tensors (BF16) from the nvidia/Qwen3.6-35B-A3B-NVFP4 model and writes them to model_mtp_head.safetensors in the AgentWorld snapshot, then creates model.safetensors.index.json pointing to both the original shard and the new MTP shard.

Run the merge script with:

python3 scripts/merge_mtp_into_agentworld.py

1. Patched config.json: Added re:^mtp.* to the quantization_config.ignore list so vLLM's compressed-tensors handler does not apply NVFP4 quantization to the MTP head, ensuring the internal parameter names match the BF16 disk tensors.

🚀 Usage with vLLM

To enable MTP speculative decoding, use the --speculative-config flag when starting vLLM:

vllm serve <model_path> \
  --tensor-parallel-size 1 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

🖥️ Hardware

Benchmarks were run on an ASUS Ascent GX10 — an NVIDIA DGX Spark system powered by the GB10 Grace Blackwell Superchip:

🐳 Docker Compose Example

qwen-agentworld:
  image: vllm/vllm-openai:nightly
  container_name: qwen-agentworld
  restart: "no"
  ports:
    - "8011:8000"
  volumes:
    - ./vllm/models:/root/.cache/huggingface
    - ./vllm/torch_compile_cache_qwen_agentworld:/root/.cache/vllm/torch_compile_cache
  environment:
    - NVIDIA_VISIBLE_DEVICES=all
    - NVIDIA_DRIVER_CAPABILITIES=compute,utility,display
    - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN:-}
    - HF_HOME=/root/.cache/huggingface
    - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
    - PYTHONUNBUFFERED=1
  command: >
    --model lgmc/Qwen-AgentWorld-35B-A3B-NVFP4-MTP
    --host 0.0.0.0
    --port 8000
    --tensor-parallel-size 1
    --gpu-memory-utilization 0.78
    --max-model-len 131072
    --max-num-seqs 64
    --max-num-batched-tokens 32768
    --quantization compressed-tensors
    --kv-cache-dtype fp8
    --moe-backend flashinfer_cutlass
    --attention-backend FLASHINFER
    --enable-chunked-prefill
    --enable-prefix-caching
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
    --trust-remote-code
    --reasoning-parser qwen3
    --tool-call-parser qwen3_coder
    --enable-auto-tool-choice
  ipc: host
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]

Start it with:

docker compose up qwen-agentworld

The model will be available at http://localhost:8011/v1 (OpenAI-compatible API).

📊 Benchmark Results

Measured with vllm bench serve on the ASUS Ascent GX10 using vllm/vllm-openai:nightly:

vllm bench serve \
  --backend vllm \
  --model lgmc/Qwen-AgentWorld-35B-A3B-NVFP4-MTP \
  --base-url http://localhost:8000 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 512 \
  --num-prompts 200 \
  --max-concurrency 32

============ Serving Benchmark Result ============
Successful requests:                     200
Failed requests:                         0
Maximum request concurrency:             32
Benchmark duration (s):                  220.58
Total input tokens:                      204800
Total generated tokens:                  102400
Request throughput (req/s):              0.91
Output token throughput (tok/s):         464.22
Peak output token throughput (tok/s):    224.00
Peak concurrent requests:                37.00
Total token throughput (tok/s):          1392.67
---------------Time to First Token----------------
Mean TTFT (ms):                          2171.01
Median TTFT (ms):                        756.89
P99 TTFT (ms):                           9620.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          62.49
Median TPOT (ms):                        57.48
P99 TPOT (ms):                           174.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           191.98
Median ITL (ms):                         170.41
P99 ITL (ms):                            410.02
---------------Speculative Decoding---------------
Acceptance rate (%):                     69.32
Acceptance length:                       3.08
Drafts:                                  33264
Draft tokens:                            99792
Accepted tokens:                         69179
Per-position acceptance (%):
  Position 0:                            78.34
  Position 1:                            68.09
  Position 2:                            61.54
==================================================

Key takeaways:

• ~464 output tok/s at 32 concurrent requests (1024 in / 512 out)
• ~60 tok/s single-request throughput (drafted throughput ≈ 62 tok/s measured independently)
• 69.3% MTP acceptance rate with average accepted length of 3.08 tokens (out of 3 speculative tokens)
• Effective per-user decode speed: ~16 tok/s (62 ms TPOT) under 32 concurrent requests

📚 References & Credits

• vLLM bug #43304 — MTP draft model inherits main model quant config (root cause)
• Reference for the MTP-embedded NVFP4 pattern: sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP
• scripts/merge_mtp_into_agentworld.py — merge script used to apply the MTP head

---

Note: This model is a derivative work based on the Qwen research and the AgentWorld fine-tune. Please ensure compliance with the original model's license (Qwen Research License or similar) before use.