NVIDIA-Nemotron-3-Super-120B-A12B-MTP-mlx

Nemotron-3-Super-120B-A12B MTP head (MLX)

The raw mtp.* tensors extracted from nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16, bf16 -- just the MTP block (mtp.layers.0 attention + mtp.layers.1 MoE), none of the target weights.

The companion mirror for the MLX engine in exo-nvidia: the speculative-decode path (SpeculativeStrategy.kind == "mtp") discovers these tensors via the mtp.* prefix, stacks the 512 per-expert weights into MLX SwitchMLP layout, and quantizes the head at load time to match the target model's precision.

Layout

mtp.layers.0.eh_proj.weight       # [hidden, 2 * hidden] -- embedding+hidden fusion
mtp.layers.0.enorm.weight         # embedding pre-norm
mtp.layers.0.hnorm.weight         # hidden pre-norm
mtp.layers.0.mixer.{q,k,v,o}_proj.weight  # attention
mtp.layers.0.norm.weight          # post-attention norm
mtp.layers.1.mixer.experts.{0..511}.{up,down}_proj.weight  # 512 MoE experts
mtp.layers.1.mixer.fc1_latent_proj.weight
mtp.layers.1.mixer.fc2_latent_proj.weight
mtp.layers.1.mixer.gate.weight
mtp.layers.1.mixer.gate.e_score_correction_bias
mtp.layers.1.mixer.shared_experts.{up,down}_proj.weight
mtp.layers.1.norm.weight             # pre-MoE norm
mtp.layers.1.final_layernorm.weight  # final norm before lm_head

mtp.layers.0 is an attention block; mtp.layers.1 is the MoE block. num_nextn_predict_layers = 1 in the upstream config.json -- the two-layer MTP head is reused autoregressively for each speculative token.