nemotron-diff-vlm-8b-int4_w4a16

Nemotron-Labs-Diffusion-VLM-8B — INT4 W4A16 (AWQ)

INT4 weight-only (W4A16) quantization of nvidia/Nemotron-Labs-Diffusion-VLM-8B, tuned for GPU-memory-constrained serving of the foot-scan classification prompt. Checkpoint is ~6.5 GiB (~6.4 GiB VRAM allocated at runtime) vs ~18 GiB BF16 for the 9B-param base.

How it was made

• NVIDIA Model Optimizer (modelopt) 0.45.0: AWQ weights-only INT4, group size 128 along the in-dim, symmetric (no zero-points), per-group weight_scale, activation smoothing via pre_quant_scale (o_proj / down_proj). Exported through compressed-tensors.
• Calibrated on 128 domain (foot-image, image-text) calibration samples. Excluded from quantization: diffusion_head, encoder.embed_tokens, encoder.multi_modal_projector*, encoder.vision_tower* (kept BF16).
• Produced and validated on a single RTX 3090 Ti (CUDA 12.6, torch 2.8.0, modelopt pinned transformers<5.10 in a separate venv).

Verification

• Weights byte-audited against the BF16 base: predicted AWQ nibbles

  = 99.5% byte-exact, 100% within +/-1 LSB per nibble.

• 48-image foot-scan set vs manual ground truth: 93.75% answer parity (BF16 baseline 95.31%), 48/48 valid JSON; answer drift vs BF16 is 7 of 192 answers, on marginal close-up views.

Runtime notes

• Loads as a standard compressed-tensors checkpoint (transformers, trust_remote_code=True). Weights are unpacked and dequantized per call (INT4 -> BF16, plain GEMM); no fused INT4 kernel. torch.compile fuses the dequant step and recovers most of the GEMM overhead.
• Known issue: transformers 5.9+ routes weights-only INT4 compressed-tensors checkpoints to the pack_quantized format, which renames weight to an uninitialized weight_packed. Loading must reject that mapping (e.g. make PackedQuantizationCompressor.can_compress return False) so the uint8 weight tensors load verbatim.

License

Derivative of the base model; use is governed by the NVIDIA Source Code License (NSCLv1) — see the base model card.