Nemotron-3.5-Lightning-1.4B-A0.1B-MTP-FP8-Dynamic

Nemotron-3.5-Lightning-1.4B-A0.1B-MTP-FP8-Dynamic

FP8 dynamic per-tensor quantized version of nvidia/NVIDIA-Nemotron-3.5-Lightning-1.4B-A0.1B-MTP using llm-compressor.

Quantization

Recipe: FP8_DYNAMIC per-tensor quantization via oneshot calibration on open-platypus (512 samples, 2048 max seq len)

MTP layers: 59 out of 61 MTP linear layers are quantized. The following are kept at full precision:

• mtp.embed_tokens (bf16) — vLLM only supports WNA16-INT for embeddings, never FP8
• mtp.eh_proj (bf16) — fusion projection is a plain nn.Linear in vLLM without quantization support

Checkpoint verification

The MTP shard (model_mtp.safetensors) should contain:

Quantized (59 linears):

• mtp.layers.*.self_attn.{q,k,v,o}_proj.weight + .weight_scale
• mtp.layers.*.mlp.{gate,up,down}_proj.weight + .weight_scale

Full precision (2 modules):

• mtp.embed_tokens.weight (bf16, no .weight_scale)
• mtp.eh_proj.weight (bf16, no .weight_scale)

The config should have a mtp_group with 59 exact targets (module names, not parameter names):

"quantization_config": {
  "config_groups": {
    "group_0": {...},
    "mtp_group": {
      "targets": [
        "mtp.layers.0.self_attn.q_proj",
        "mtp.layers.0.self_attn.k_proj",
        ...
      ],
      "weights": {"num_bits": 8, "type": "float", "strategy": "tensor", ...}
    }
  }
}

What to check: If mtp.eh_proj.weight_scale or mtp.embed_tokens.weight_scale exist, the checkpoint is broken — vLLM will load the FP8 weight without the scale and produce corrupted outputs.

Serving

vllm serve soyrsoyr/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP-FP8-Dynamic --max-model-len 4096

Performance

Quantized checkpoint size: ~1.5GB (vs ~2.8GB unquantized)