nemotron-super-120b-cc-mt-uniform-dr

nemotron-super-120b-cc-mt-uniform-dr

Nemotron 3 Super 120B-A12B continued-pretraining (midtraining) checkpoint from the constitutional-curriculum midtraining study (cc-mt).

Training

• Base checkpoint: NVIDIA-Nemotron-3-Super-120B-A12B-Base-Chat-Init-BF16 (pure Base weights; 1,188 chat-scaffolding embedding rows grafted from Instruct. No post-training behavior included.)
• Data: 1:1 blend of cho-ai/constitutional-curriculum-mt-data data/uniform_DR.jsonl (506.9M tokens, pre-shuffled, no phase ordering) and shuffled pretraining replay (geodesic-research/Nemotron-Pretraining-Specialized)
• Recipe ("dyad-1" Base-CPT stability stack): lr 1e-6 cosine w/ 10% warmup, FP32 optimizer states, GBS 128, seq 8192, BF16, 967 iterations (~1.0B tokens, 1 epoch). Final lm loss 0.762 on the final batch (single-batch noise; ~0.57-0.64 over the last ~200 iterations, matching the July run). Tokenizer: geodesic-research/nemotron-base-tokenizer (EOD=</s>=id 2).
• Topology: TP=1 EP=4 PP=22 ETP=1 (parallel folding), 22 nodes / 88 GH200 GPUs on Isambard-AI.
• W&B: geodesic/megatron_training/zlxcuwud

Provenance note (rerun, 2026-10)

The original July 2026 upload of this repo was lost together with its local checkpoints. This model is a retrained replicate: the identical config (verified field-by-field against the original run's recorded config), code (geodesic-megatron fef73653, Megatron-LM 3758b54b), environment, data, seed and data order, re-run in October 2026. Megatron MoE training is not bitwise-deterministic, so the weights are statistically equivalent to, but not identical with, the lost July weights.

Usage notes

Base-style model (pretraining-format CPT; no instruction tuning) — use completion-style prompting. Loads with native transformers NemotronHForCausalLM; tokenizer PreTrainedTokenizerFast. Coherence check (completion mode): W&B ntgw6u06.

SFT companions: nemotron-super-120b-cc-mt-uniform-dr-sr-sft (reasoning) and nemotron-super-120b-cc-mt-uniform-dr-200k-sft (non-reasoning).