Qwen3.5-27B-Text-NVFP4-MTP

Qwen3.5-27B-NVFP4-MTP

NVFP4-quantized Qwen/Qwen3.5-27B with the original bf16 MTP (Multi-Token Prediction) head grafted back in for speculative decoding. Achieves a 3.3x speedup on a single DGX Spark over bf16 with TP=2 on two Sparks.

Important flags

• --language-model-only — Required. Qwen3.5 is natively multimodal; this skips the vision encoder (no vision weights are included).
• --quantization modelopt — Required. Tells vLLM to use the NVFP4 weight format.
• num_speculative_tokens: 1 — The model has a single MTP layer. Setting this to 2 reuses the same layer twice with degraded acceptance rates.

What's Different About This Quant

Standard NVFP4 quantization via nvidia-modelopt discards the MTP head because AutoModelForCausalLM doesn't load it. This checkpoint restores the MTP weights from the original model in bf16 and adds them to the quantization ignore list, giving you working speculative decoding out of the box.

Quantization Details

Usage (vLLM)

Requires vLLM with Qwen3.5 support (tested with vllm/vllm-openai:qwen3_5-cu130, vLLM 0.16.0rc2).

With MTP speculative decoding (recommended)

vllm serve osoleve/Qwen3.5-27B-NVFP4-MTP \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --quantization modelopt \
    --language-model-only \
    --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Without MTP (still faster than bf16)

vllm serve osoleve/Qwen3.5-27B-NVFP4-MTP \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --quantization modelopt \
    --language-model-only

Disabling thinking mode

Qwen3.5 defaults to thinking/reasoning mode. To disable it per-request:

{
    "chat_template_kwargs": {"enable_thinking": false}
}

Benchmarks

Tested on NVIDIA DGX Spark (GB10 SoC, 128 GB unified CPU/GPU memory). Task: generate 692 tokens with thinking disabled.

Memory footprint: ~18.5 GiB with MTP, leaving ~80 GiB for KV cache on a 128 GB system.

How This Was Made

1. Loaded Qwen/Qwen3.5-27B in bf16 via AutoModelForCausalLM
2. Quantized to NVFP4 using modelopt.torch.quantization.quantize() with NVFP4_DEFAULT_CFG
3. Exported via modelopt.torch.export.unified_export_hf.export_hf_checkpoint()
4. Grafted 15 MTP tensors from the original checkpoint into the quantized safetensors
5. Added MTP linear layers to quantization_config.ignore in config.json
6. Used the original model's config.json structure (ForConditionalGeneration with text_config/vision_config) to match weight naming conventions

License

Apache 2.0, following the original Qwen3.5-27B license.