AdaptKey-Nemotron-30b-FP8

AdaptKey-Nemotron-30b-FP8

FP8 Dynamic quantization of AdaptKey/AdaptKey-Nemotron-30b (hybrid Mamba2 + MoE).

Quantization

• Scheme: FP8_DYNAMIC (W8A8, per-channel weights, dynamic per-token activations)
• Serialization: compressed-tensors
• Kept at higher precision: MoE routers (mixer.gate), Mamba2 state params (A_log, D, dt_bias, conv1d), all norms, embeddings, lm_head, the attention projections and the Mamba projections immediately preceding each attention block.

Serving with vLLM

config.json declares auto_map -> configuration_nemotron_h.NemotronHConfig and the file ships with the checkpoint, so --trust-remote-code is required for transformers to parse the config. vLLM then uses its own native NemotronHForCausalLM implementation for the weights.

vllm serve ukkathva/AdaptKey-Nemotron-30b-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.90

On NVIDIA Ampere (A100, sm_80) there is no native FP8 tensor-core path; vLLM serves FP8 MoE via its MARLIN backend, so the benefit is memory, not throughput.