Qwen3-Coder-30B-A3B-NVFP4

Qwen3-Coder-30B-A3B-NVFP4

NVFP4 quantization of Qwen3-Coder-30B-A3B-Instruct. 61.08 GB down to 18.11 GB (3.37x), and on a single B200 it is faster than FP8 at every concurrency level measured, with no detectable loss on HumanEval.

This is the third member of a three-way release built on the same base model. Its siblings are Qwen3-Coder-30B-A3B-W4A16 and Qwen3-Coder-30B-A3B-Prune3-W4A16. The three together make the point of the release: NVFP4 and W4A16 are both four bit formats, and they land on opposite sides of FP8. W4A16 stores 4-bit weights and dequantizes them at compute time, so it buys memory and pays compute. NVFP4 quantizes activations as well as weights and runs natively on the Blackwell FP4 tensor cores, so on that hardware it buys memory and speed at the same time.

Quantization recipe

Quantized with llm-compressor using GPTQ with scheme=NVFP4. The MoE router is kept at high precision by excluding it from quantization with the pattern re:.*mlp.gate$, so only the expert weights and activations go to FP4. Calibration used 1024 samples at 2048 sequence length. On-disk format is compressed-tensors, quantization method nvfp4-pack-quantized.

The four-way comparison

All four arms below are the same model, Qwen3-Coder-30B-A3B-Instruct, served with vLLM 0.27.1 on a single B200 (tkai-dev-compute-b200, driver 580.65.06), with VLLM_USE_DEEP_GEMM=0 and VLLM_ATTENTION_BACKEND=TRITON_ATTN. Workload: unique prompt per request, input 1746 tokens actual (2048 token target), output 256 tokens, ignore_eos, temperature 0, streaming, 3 repetitions per concurrency level. The arms ran serially on the same node to hold whatever background load existed constant across them. Measured 2026-08-13.

Output throughput (tokens per second)

NVFP4 beats FP8 at every concurrency measured, from 1.07x at concurrency 256 up to 1.32x at concurrency 8. It beats W4A16 by a wider margin throughout, 1.40x to 1.83x. Throughput rolls over between concurrency 128 and 256 for all four arms, so the 256 and 512 rows are past the saturation point and their ratios compress toward 1.0. Read concurrency 128 as the peak-efficiency reading, not the ceiling.

Energy efficiency (tokens per joule)

At concurrency 128 the efficiency gap is wider than the throughput gap: NVFP4 reaches 12.82 tokens per joule against FP8's 9.48, a 1.35x advantage.

Code quality: HumanEval pass@1 (n=164)

The spread across all four arms is 2.44 percentage points, smaller than a single arm's own stderr of about 2.17 points. NVFP4 against the bf16 baseline gives z = -0.58, not a significant difference. NVFP4 is numerically the lowest score of the four, and that is reported as measured rather than rounded away. The honest claim this test supports is no detectable loss relative to bf16, nothing stronger. It does not show NVFP4 matching or beating the others, only that a 164-question sample cannot separate any of the four.

Harness: lm-eval 0.4.12, task humaneval_instruct, --apply_chat_template, --gen_kwargs continue_final_message=True,add_generation_prompt=False, HF_ALLOW_CODE_EVAL=1, served through local-chat-completions against the same vLLM 0.27.1 instance used for the throughput sweep. Both the env var and the gen_kwargs are load bearing: without the env var the task fails to import, and without the gen_kwargs every arm, including the bf16 control, scores 0.0, which is how a misconfiguration was caught during this run. As a sanity check on the harness itself, the bf16 baseline here (0.9207) was compared against a bf16 run of the same eval from a 2026-08-06 ledger (0.9267): 0.60 percentage points apart, inside normal run-to-run noise for a 164-question sample.

Kernel path: verified, not assumed

Earlier NVFP4 numbers in this repo were read off logs after the run, never asserted at run time. For this comparison the harness records the MoE backend name and a fallback-warning flag for each arm and only accepts a throughput number once that check passes.

Only the nvfp4 arm reports an FP4 kernel path, and it is the native one: FLASHINFER_TRTLLM, zero fallback warnings, on sm100 (Blackwell). The other three arms legitimately skip the FP4 gate, since they are not FP4 formats. This is the check every NVFP4-versus-something claim needs and usually does not carry: proof that the number came from the FP4 tensor cores and not from an emulated path.

Hardware requirement: Blackwell only

NVFP4 needs the FP4 tensor cores present on Blackwell (B200, and consumer Blackwell parts) and on nothing older. On H200 or H100 (Hopper), vLLM has no native FP4 path and falls back to weight-only Marlin emulation, measured separately at 0.85x of the equivalent FP8 throughput on that hardware. This checkpoint will load and run on Hopper, but it will not deliver the numbers on this card there. For Hopper deployments, W4A16 is the more appropriate 4-bit format.

Serving with vLLM

Requires vLLM 0.27.1 or newer on a Blackwell GPU (sm100), such as a B200. The checkpoint is compressed-tensors format and vLLM reads the NVFP4 quantization config directly from it:

vllm serve ThakiCloud/Qwen3-Coder-30B-A3B-NVFP4 \
  --quantization compressed-tensors

Confirm the native path is in use by checking the startup logs for the FLASHINFER_TRTLLM MoE backend selection and the absence of any FP4 fallback warning. A fallback warning means either the GPU is not Blackwell or vLLM could not select the native kernel, and the throughput numbers above will not apply.

Caveats

• One model family, one GPU generation, one workload shape (1746 input tokens, 256 output tokens). No claim is made outside that shape.
• The node contention snapshot only sees the GPU exposed to this pod, so it records our own device and cannot prove the other seven GPUs on the node were idle. Running the four arms serially on the same node keeps whatever background load existed constant across them, which does not eliminate the concern but does hold it fixed.
• Throughput rolls over between concurrency 128 and 256 for every arm; the 256 and 512 rows are past saturation and their ratios compress toward 1.0.
• HumanEval at n=164 cannot statistically separate any of the four arms. Read the quality result as "no detectable loss," not as "better" or "equal."
• NVFP4 is Blackwell-only. It falls back to Marlin emulation on Hopper (H200, H100), measured at 0.85x, and none of the throughput or energy numbers above transfer to that hardware.

Provenance

Quantized by ThakiCloud with llm-compressor. Base model Qwen/Qwen3-Coder-30B-A3B-Instruct is Apache-2.0 and not gated; this derivative inherits that license. Quantization recipe and calibration configuration are in recipe.yaml in this repo. Full measurement ledger: whitepapers/data/ledger/2026-08-13-nvfp4-vs-fp8-coder-fourway.json in the ThakiCloud/ai-platform-strategy internal repo, with raw per-arm throughput and HumanEval data under whitepapers/data/ledger/raw/2026-08-13-nvfp4-vs-fp8-coder/.