Qwen3-4B-Instruct-NVFP4

Qwen3-4B-Instruct-NVFP4

Overview

This repository provides an NVFP4 (FP4 E2M1) mixed-precision quantized build of Qwen3-4B-Instruct-2507 in ComfyUI comfy_quant format, primarily intended for use as a text encoder (e.g. for Z-Image).

Many thanks to SergiusFlavius, who helped me generate this quantization.

Quantization Details

• Backend: ComfyUI convert_to_quant v1.2.6 (comfy_kitchen CUDA NVFP4 kernels)
• Format: comfy_quant mixed precision
  • NVFP4 (FP4 E2M1, 16-element blocks, learned rounding) — text transformer blocks 2–33 projections only
  • FP8 (float8_e4m3fn, tensorwise, learned rounding) — text blocks 1 & 34
  • FP16 (bfloat16) — embed_tokens, model.norm, all norms/biases, pos_embed, patch_embed, text blocks 0 & 35, and the entire vision encoder (blocks, merger, deepstack_merger_list)
• No input_scale: ComfyUI quantizes activations dynamically at runtime (per-tensor amax via the NVFP4 layout), so a baked-in activation scale is not required. This matches the stock Qwen3-VL NVFP4 baseline.
• Hardware target: NVIDIA Blackwell (RTX 50 series, SM 12.0) — required for native NVFP4 tensor-core execution (see Runtime below).
• Size: 4.03 GB / 1413 tensors (vs 8.88 GB FP16 source — ~55% smaller).

Layer breakdown

Runtime / hardware

How the layers execute depends on the GPU (ComfyUI pick_operations gates each format on the device capability):

On Blackwell, NVFP4 weights stay packed and run through comfy_kitchen's TensorCoreNVFP4Layout GEMM (FP4 weight × dynamically-quantized FP8 activation). On non-Blackwell GPUs the format is emulated via dequantization, so you get the smaller footprint but not the FP4 speedup.

Usage (ComfyUI)

Place qwen3_vl_4b_instruct_nvfp4.safetensors in ComfyUI/models/text_encoders/.

• As Z-Image text encoder: load with a CLIPLoader node, type lumina2.

ComfyUI auto-detects the quantization metadata (Found quantization metadata version 1) and selects MixedPrecisionOps for the text encoder.

License

Apache-2.0 (inherited from the base model).