Qwen3.8-Flash-Next-Nota-NVFP4

Nota Qwen3.8-Flash-Next · NVFP4 (W4A4)

A 4-bit (NVFP4) quantized release of Qwen's Qwen3.8-Flash-Next — a 180B-parameter natively multimodal Mixture-of-Experts model with ~7B active per token, built on the architecture that will underpin Qwen4.

<div align="center">

Half the GPUs, in every configuration

335.3 GiB → 173.6 GiB (51.8%)

Full 262,144-token context · MTP speculative decoding preserved

</div>

Minimum cards that serve one full 262,144-token request, from measured per-rank footprints.

Highlights

• NVFP4 (4-bit float, W4A4) — group_size=16, packed in the compressed-tensors nvfp4-pack-quantized format for direct serving in vLLM. Both weights and activations are quantized to 4-bit floating point.

  • Requires NVIDIA Blackwell. NVFP4 relies on the FP4 tensor cores introduced in the Blackwell architecture (e.g. B200 / B300 / GB200 / RTX PRO 6000). Earlier architectures (Hopper, Ada, Ampere) do not support NVFP4 execution.

• Only the routed experts are quantized. They hold 67.1% of the parameters and 84.2% of the BF16 checkpoint's bytes, so the memory saving is captured almost in full while every precision-critical path stays in BF16 — the same split the reference NVFP4 release uses.
• No architecture change. Layer count, expert count and expert routing are identical to the base checkpoint, so stock vLLM serves it as-is — no patched modeling file.
• MTP, PLE and multimodal preserved. The multi-token-prediction block, the N-gram PLE embedding and the vision tower stay BF16, so speculative decoding, the PLE pathway and image/video inputs work as in the base model.

Performance

• Total context budget 131,072 tokens; generation length 65,536 tokens.

• Greedy decoding (temperature=0, top_p=1.0, seed=0), single pass. These evaluations are intended to quantify quantization degradation rather than to produce leaderboard figures, and therefore employ no sampling, no avg@n and no self-consistency.

• HRM8K is restricted to the KSM subset (1,428 of 8,011 items). KSM is the only subset drawn from original Korean mathematics examinations; the remaining four subsets (MATH, GSM8K, OMNI_MATH, MMMLU) are translations of English benchmarks and would duplicate the reasoning already assessed by AIME and GPQA.

• BFCL v4 is restricted to the 13 single-turn categories: simple_python, simple_java, simple_javascript, multiple, parallel, parallel_multiple, irrelevance, live_simple, live_multiple, live_parallel, live_parallel_multiple, live_irrelevance and live_relevance. The multi-turn and memory categories are excluded, as are the two web-search categories, which require an external search API key.

• Where NVFP4 exceeds BF16, the result should be interpreted as an absence of degradation rather than an improvement. Each difference falls within the sampling error of its benchmark. Four-bit quantization does not improve a model; under greedy decoding, minute numerical differences resolve individual questions in either direction, and the aggregate may fall marginally above or below the baseline.

Quantization scope

The 73,728 quantized modules are 512 experts × 48 layers × 3 projections (gate_proj, up_proj, down_proj). The base checkpoint stores them as 96 fused 3-D tensors (mlp.experts.gate_up_proj [512, 1280, 2560], mlp.experts.down_proj [512, 2560, 640]); this release unfolds them into per-expert mlp.experts.<e>.<proj> modules, which is what compressed-tensors and vLLM's RoutedExperts loader expect.

Kept in BF16:

Where the bytes are

The PLE N-gram embedding is a 320,001,536 × 160 hashed-trigram lookup table — 28.5% of the parameters at almost zero compute per parameter. It is left in BF16 because quantizing a table that is read, not multiplied, buys nothing in speed; instead it is the natural candidate for host memory (see below), which is what brings this model down to two GPUs.

Calibration

512 conversations of exactly 4,096 tokens, rendered through the Qwen chat template and drawn from the workloads this model is built for rather than generic web text:

27 sources in total. 65.6% of the samples carry reasoning traces inside <think> blocks and 38.9% contain <tool_call> / <tool_response> turns, so the calibration activations cover the routing patterns of reasoning and tool use rather than plain prose.

Requirements

vllm      built with qwen4_exp support   (vllm/vllm-openai:qwen38-flash-next or newer)
flashinfer >= 0.6.17

Pin the MoE kernel. vLLM's automatic NVFP4 MoE backend selection picks FLASHINFER_TRTLLM, which silently corrupts decode on this model: requests return HTTP 200 and the output degenerates into a single repeated token. This is not specific to this release — the reference NVFP4 checkpoint fails identically. Always pass --moe-backend flashinfer_cutlass (Blackwell datacenter, sm100/sm103) or --moe-backend marlin (RTX PRO 6000 / GB202, sm120).

Quick Start

RTX PRO 6000 × 2 — PLE in host memory

VLLM_PLE_CPU_OFFLOAD=1 CUDA_VISIBLE_DEVICES=0,1 \
vllm serve nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --served-model-name nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --tensor-parallel-size 2 \
  --moe-backend marlin \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice

B200 × 2 — PLE resident on GPU

CUDA_VISIBLE_DEVICES=0,1 \
vllm serve nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --served-model-name nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --tensor-parallel-size 2 \
  --moe-backend flashinfer_cutlass \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice

RTX PRO 6000 × 4 — add --enable-expert-parallel

CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --served-model-name nota-ai/Qwen3.8-Flash-Next-Nota-NVFP4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend marlin \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --no-enable-flashinfer-autotune \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice

--enable-expert-parallel is required at TP=4, not optional. moe_intermediate_size is 640 = 2⁷ × 5, so tensor parallelism shards it to 160 per rank and the NVFP4 MoE kernels reject the alignment (flashinfer_cutlass: "Intermediate size padding for w1 and w3 … not currently supported"; flashinfer_cutedsl: "dim 1 size 320 must be divisible by 128"). Expert parallelism splits the MoE by expert instead, leaving moe_intermediate_size at 640 on every rank; 512 experts divide evenly by 1, 2, 4 and 8.

Do not add --enable-eplb. Qwen3.8-Flash-Next's reasoning_effort parameter (low / medium / xhigh) works unchanged.