nemotron-super-120b-cc-mt-curriculum-dr

nemotron-super-120b-cc-mt-curriculum-dr

Nemotron 3 Super 120B-A12B continued-pretraining (midtraining) checkpoint from the constitutional-curriculum midtraining study (cc-mt).

Training

• Base checkpoint: NVIDIA-Nemotron-3-Super-120B-A12B-Base-Chat-Init-BF16 (pure Base weights; 1,188 chat-scaffolding embedding rows grafted from Instruct. No post-training behavior included.)
• Data: 1:1 blend of cho-ai/constitutional-curriculum-mt-data data/curriculum_DR.jsonl (506.9M tokens, consumed in exact file order: value-cluster phases 1-4) and shuffled pretraining replay (geodesic-research/Nemotron-Pretraining-Specialized)
• Data order: FILE-ORDER curriculum (phases 1-4 sequential; boundaries at iters 242/483/725)
• Recipe ("dyad-1" Base-CPT stability stack): lr 1e-6 cosine w/ 10% warmup, FP32 optimizer states, GBS 128, seq 8192, BF16, 967 iterations = 1.014B tokens (1 epoch). Tokenizer: geodesic-research/nemotron-base-tokenizer (EOD=</s>=id 2). All data filtered against the 1,188 zero-embedding token ids of Super-Base (0 docs dropped).
• Topology: TP=1 EP=4 PP=22 ETP=1 (parallel folding), 22 nodes / 88 GH200 GPUs on Isambard-AI.
• W&B: geodesic/megatron_training/uh70x0w1

Usage notes

Base-style model (pretraining-format CPT; no instruction tuning) — use completion-style prompting. Loads with native transformers NemotronHForCausalLM; tokenizer PreTrainedTokenizerFast. Coherence-checked post-export (8/8 non-empty, fluent completions).

SFT companions: nemotron-super-120b-cc-mt-curriculum-dr-sr-sft (reasoning) and nemotron-super-120b-cc-mt-curriculum-dr-200k-sft (non-reasoning).