Qwen3-8B-NVFP4

Qwen3-8B-NVFP4

Qwen3-8B (pinned revision b968826) quantized with NVFP4 (llm-compressor @ git main, W4A4: 4-bit weights AND activations, Blackwell-native block scaling), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with frozen splits, capability probes, and bootstrap 95% CIs on an RTX 5090. Write-up: "NVFP4 on a 5090 names two different machines".

Most quantized checkpoints ship with no damage documentation. This card tells you what this quantization costs, per capability, against the BF16 baseline measured on identical hardware with identical frozen evals.

Damage pools in computation, not structure. Full five-variant table with bootstrap CIs: the study.

Every number traces to a results JSON in the repo, and every eval run recorded which GPU kernel actually served it. This checkpoint's serving receipt on our stack (vLLM 0.26.0, sm_120):

Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM

Serving this on consumer Blackwell (sm_120) is nontrivial; read before deploying.

• The native FP4 path JIT-compiles flashinfer's fp4_gemm_cutlass_sm120 at first load. On a conda toolchain you may need curand headers on the include path, MAX_JOBS=4, and LIBRARY_PATH=/usr/lib/x86_64-linux-gnu (full chain: postmortem).
• First-contact fp4_gemm autotune costs HOURS on sm_120 (per capture-size x layer-shape, ~1.5 min each; flashinfer#4110 data). VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=fp4_gemm skips it (accuracy unchanged, speed default-tactic).
• vLLM may route this checkpoint through the Marlin weight-only kernel instead of native FP4 depending on stack and flags, with different speed and slightly different outputs. Check your serving log for the kernel line before trusting any measurement. Both paths measured side by side in the repo (--linear-backend marlin forces the fallback; it was FASTER single-stream at default tactics).

Part of sm120-quant-bench

One of three checkpoints quantized and evaluated together in sm120-quant-bench; siblings: Qwen3-8B-W4A16-GPTQ and Qwen3-8B-W4A16-AWQ.

Studies using these checkpoints

• NVFP4 on a 5090 names two different machines
• Your agent's tools survive 4-bit. Its token budget and its defaults don't.

How it was made

recipes/nvfp4.py (llm-compressor from git main: the released 0.12 line hits issue #3011's misleading OOM without max_seq_length; 128 samples, max_seq 2048, seed 3407). Calibration data is disjoint from every eval split by construction. Full recipe, harness, and raw results: the repo. Reproduction gets byte-identical eval items (seed-pinned, hashed manifest).

Serve

vllm serve <this-repo> --max-model-len 16384

Check the log for the kernel-selection line before benchmarking. If a number matters, its serving receipt matters.