Downloads · 30 days
44
18% of all-time downloads
JongYeop/Qwen3-30B-A3B-NVFP4-W4A4
Qwen3-30B-A3B-NVFP4-W4A4 is a text generation model from JongYeop. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Qwen/Qwen3-30B-A3B quantized to NVFP4 W4A4 (NVIDIA FP4, 4-bit weights + 4-bit activations) with llm-compressor.
Downloads · 30 days
44
18% of all-time downloads
All-time downloads
242
Public
Parameters
17.5B
18.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.1 GB · 100%
How the weights are stored.
U814.9B · 86%
From the Hugging Face model README
Qwen/Qwen3-30B-A3B quantized to NVFP4 W4A4 (NVIDIA FP4, 4-bit weights + 4-bit activations) with llm-compressor.
This is a Mixture-of-Experts (MoE) model. The MoE router (mlp.gate) and the
lm_head are kept at full precision; the attention and expert projection layers
are quantized to NVFP4.
lm_head, MoE router (re:.*mlp.gate$)HuggingFaceH4/ultrachat_200k
(train_sft), max sequence length 2048NVFP4 fused-MoE inference requires a runtime kernel for your GPU. The cutlass
NVFP4 grouped-GEMM MoE kernel (get_cutlass_moe_mm_data in vLLM) is currently
compiled only for CUDA compute capability 9.0 (Hopper) and 10.0
(datacenter Blackwell, B200/GB200).
On SM120 GPUs (e.g. RTX PRO 6000 Blackwell, compute capability 12.0), the
stock vLLM build used to produce this checkpoint does not yet ship a compiled
NVFP4 MoE kernel, so vLLM serving fails with
No compiled get_cutlass_moe_mm_data: ... capability 120. Required capability: 90 or 100.
This is a runtime kernel limitation, not a problem with the checkpoint — the
quantized weights were verified (correct NVFP4 packing; router and lm_head left
in full precision). To serve this model, use a GPU/kernel combination that
provides an NVFP4 fused-MoE kernel (Hopper or B200), or a vLLM build with NVFP4
MoE kernels compiled for your architecture.
from vllm import LLM, SamplingParams
llm = LLM(model="JongYeop/Qwen3-30B-A3B-NVFP4-W4A4")
out = llm.generate(
["Explain mixture-of-experts in one sentence."],
SamplingParams(temperature=0.7, max_tokens=128),
)
print(out[0].outputs[0].text)
quant_stage:
quant_modifiers:
QuantizationModifier:
ignore: ["lm_head", "re:.*mlp.gate$"]
scheme: "NVFP4"
targets: ["Linear"]