Qwen3-8B-NVFP4-W4A16

Qwen3-8B-NVFP4-W4A16

Weight-only NVFP4 (W4A16) derivative of raoashish10/Qwen3-8B-NVFP4, produced specifically for serving stacks that can't run that checkpoint's real W4A4 (weights and activations) quantization — most notably vLLM on consumer Blackwell (SM120, e.g. RTX 50-series) GPUs, where vLLM's modelopt_fp4 loader fails to load the W4A4 checkpoint outright (FlashInfer has no SM120 build of the cutlass kernel it needs; see the warning on the source repo).

If you're serving with TensorRT-LLM or SGLang, you almost certainly want raoashish10/Qwen3-8B-NVFP4 instead — that's the real, fully-calibrated W4A4 checkpoint, and both those backends serve it correctly. This repo exists only to unblock vLLM on hardware that can't run the W4A4 path.

What this is

Derived directly from Qwen3-8B-NVFP4 by:

1. Relabeling quant_algo from NVFP4 to W4A16_NVFP4 in both config.json and hf_quant_config.json, and removing the input_activations entry from config.json's quantization_config.config_groups.group_0 (there's no activation quantization here, so nothing should claim there is).
2. Stripping the 252 per-layer input_scale tensors from model.safetensors — these were real, calibrated activation-scale factors from the source checkpoint's W4A4 quantization, but they're inert in a weight-only serving path and would misrepresent the checkpoint if left in place next to a W4A16_NVFP4 label.

The 4-bit weight tensors themselves (.weight, .weight_scale, .weight_scale_2) are untouched — byte-identical to the source repo. NVIDIA TensorRT-Model-Optimizer's weight quantizer is calibrated identically regardless of whether activation quantization is also enabled (both the W4A4 and W4A16 presets import the same weight-quantizer config), so this is a subtractive-only change: nothing was re-quantized, nothing about the weights changed, only the now-inapplicable activation-quantization metadata and tensors were removed.

Serving with vLLM

No metadata patch needed — this repo is correctly labeled, so vLLM's modelopt_fp4 loader picks the weight-only (Marlin) path on its own:

vllm serve raoashish10/Qwen3-8B-NVFP4-W4A16 --quantization modelopt_fp4

Verified on an RTX 5070 Ti (SM120) with vLLM 0.28.0: loads via MarlinNvFp4LinearKernel and produces coherent output, no manual quant_algo edit required. This checkpoint ships with Qwen3's "thinking mode" on by default — pass chat_template_kwargs={"enable_thinking": false} in chat-completion requests for low-latency use.

License

Apache-2.0, inherited from the base Qwen/Qwen3-8B model, same as the source Qwen3-8B-NVFP4 repo.