Qwen3.5-4B-NVFP4

LostGentoo/Qwen3.5-4B-NVFP4

Trusted, reproducible NVFP4 quantization of Qwen/Qwen3.5-4B (Apache-2.0) produced with NVIDIA ModelOpt 0.45 for Blackwell inference (vLLM quantization=modelopt_fp4).

This is a text-chat oriented build of the official VLM wrapper (Qwen3_5ForConditionalGeneration). The vision tower is left unquantized / unused at serve time — send text only.

Why this exists

This release exists to provide a known base + auditable calibration for Qwen3.5-4B in ModelOpt NVFP4 form. It publishes the exact preset, public calibration mix, seed, hashes, and measured speed/quality so the recipe can be re-run independently.

Recipe

Scripts live with the producer workspace (build_calib.py, quantize_modelopt.py). AWQ_LITE was attempted but export hit a Qwen fuse/pre_quant_scale shape error; DEFAULT (max) is the published path.

Serve (vLLM ≥ 0.25 recommended; tested 0.26.0)

python -m vllm.entrypoints.openai.api_server \
  --model LostGentoo/Qwen3.5-4B-NVFP4 \
  --served-model-name LostGentoo/Qwen3.5-4B-NVFP4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

vLLM auto-detects ModelOpt NVFP4 (quantization=modelopt_fp4). On Blackwell you should see:

Using CutlassNvFp4LinearKernel for NVFP4 GEMM

Hardware: native FP4 GEMM requires NVIDIA Blackwell. Non-Blackwell will not get the intended speedup.

Tip: Qwen3.5 may emit a “Thinking Process” preamble. Prefer higher max_tokens for evals, or disable thinking in the chat template if your stack supports enable_thinking=false / /no_think.

Measured — gentoo 1× RTX 5060 Ti (SM120), vLLM 0.26.0

Server-side deltas via vllm_probe.py delta + ctxsweep / prefill_probe (unique prompts).

Decode aggregate @6144 ctx

Whole-matrix server delta decode ≈ 334 tok/s. Prefill peak @6144 ≈ 17.1k tok/s (conc1 fit ≈ 16.2k). 0 preemptions / sweep cell errors. Kernel: CutlassNvFp4LinearKernel.

Quality (thinking disabled where noted)

Cloud Run RTX PRO 6000 spot

Service qwen35-4b-axion-nvfp4-b6000, vLLM v0.24.0 image + stdlib proxy, max-num-seqs=128. Native fp4_gemm.

Prefill peak @6144: 69.7k tok/s. 0 errors / preemptions. Scaled to minScale=0 after the spot.

5060 Ti remains a consumer SM120 signal; B6000 is the G4-class capacity number.

Files

• model.safetensors — NVFP4 weights (+ unquantized excluded modules)
• hf_quant_config.json — ModelOpt producer metadata
• config.json — architectures: [Qwen3_5ForConditionalGeneration], quantization_config.quant_algo: NVFP4
• Tokenizer / processor / chat template from the base model

Limitations

• VLM architecture quirk: not a pure ForCausalLM checkpoint (vLLM text-only path still works).
• Public UltraChat/FineWeb calib — not domain-tuned for any private corpus.
• Default thinking mode can burn completion budget; configure your client accordingly.
• Consumer Blackwell ≠ datacenter B6000 bandwidth; re-bench on target SKU before capacity planning.

Acknowledgements

• Base model: Qwen team (Qwen/Qwen3.5-4B)
• Quantization: NVIDIA ModelOpt
• Structural recipe validated against vLLM loading behavior and the exported hf_quant_config.json layout

Citation

@misc{lostgentoo_qwen35_4b_nvfp4,
  title  = {Qwen3.5-4B NVFP4 (ModelOpt, public calib)},
  author = {LostGentoo},
  year   = {2026},
  url    = {https://huggingface.co/LostGentoo/Qwen3.5-4B-NVFP4}
}