smolmo-32b-nvfp4-step12750

smolmo-32b-nvfp4-step12750

NVFP4 (W4A4) quantization of chankhavu/olmo_32b_fp8_niiattempt2_step12750_20260617055513, a 32B dense Olmo3ForCausalLM SFT checkpoint, produced with NVIDIA TensorRT Model Optimizer (ModelOpt) 0.44.0.

Quantization

• Format: NVFP4 E2M1 with block scaling — group_size=16, FP8 E4M3 per-block scale + FP32 per-tensor scale, on both weights and activations (W4A4).
• Kept in BF16: lm_head; and, under the windowed attention recipe, the q/k/v/o projections of every full-attention layer and the windowed layer immediately preceding each full layer (Olmo3's 3:1 sliding:full pattern). All MLP layers are quantized to NVFP4.
• KV cache: not quantized (kept BF16). For runtime FP8 KV cache use vLLM's --kv-cache-dtype fp8.

This selective recipe protects the long-context global-attention path (the most sensitive component for reasoning) while quantizing the bulk of the parameters. Native FP4 compute requires Blackwell (SM100+, incl. RTX 6000 Pro / sm_120); older GPUs fall back to weight-only dequant.

Calibration

• Data: chankhavu/smolmo-olmo3-calib-4k, all ~4000 samples.
• Sequence length: 16384 tokens per sequence (truncated to natural length, no padding).
• Algorithm: max (naive per-block absmax — NVFP4_DEFAULT_CFG).
• Procedure: data-parallel calibration across 4 GPUs (one full replica per GPU, each over a disjoint shard of the data); amax is merged across ranks by elementwise max — equivalent to single-process calibration over the full set.

Usage

Loads natively in vLLM and SGLang:

from vllm import LLM
llm = LLM(model="chankhavu/smolmo-32b-nvfp4-step12750")

NVFP4 compute requires a Blackwell GPU (SM100+); other GPUs dequantize to a higher precision for compute.

Notes

Exported as a unified Hugging Face checkpoint; quantization details are recorded in hf_quant_config.json. This is PTQ only (no quantization-aware training). 4-bit quantization will trail FP8/BF16 somewhat on hard math/reasoning benchmarks — evaluate before relying on it.