nemotron-climb-proxy-models

Model Overview

Description:

Nemotron-CLIMB Proxy Base Models (62M and 350M) are two small decoder-only transformer language models pre-trained from scratch by NVIDIA on 10 trillion tokens using the Megatron-LM codebase. They are designed as proxy models for scaling law research — enabling practitioners to forecast the behavior of much larger models prior to committing full-scale compute resources. Both models use a WSD (Warmup-Stable-Decay) learning rate schedule and share the same 32-layer architecture, differing only in hidden dimension. These models are ready for commercial/non-commercial use. <br>

License/Terms of Use:

Released under the NVIDIA Open Model License. <br>

Deployment Geography:

Global <br>

Use Case: <br>

These proxy models are intended for ML researchers and engineers working on:

• Scaling law experiments — predicting loss, downstream accuracy, or emergent behavior of larger models from small-model trends.
• Recipe transfer — validating hyperparameter choices (learning rate, batch size, data mix) at low cost before scaling up.
• Proxy-tuning research — studying how fine-tuning dynamics (SFT, RLHF, DPO) transfer across model scales.
• Reward model proxy training — training lightweight reward models for alignment research. <br>

References(s):

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism <br>
• Scaling Data-Constrained Language Models <br>
• Scaling Laws for Neural Language Models <br>
• DCLM: DataComp for Language Models <br>

Model Architecture:

Architecture Type: Transformer (decoder-only) <br>

Network Architecture: Decoder-only transformer with RMSNorm, SwiGLU activation, and Rotary Position Embeddings (RoPE). <br>

Number of model parameters: <br>

Note: Checkpoint sizes include optimizer state and RNG state, suitable for continued pre-training. <br>

Design Choices: Both models were trained from scratch using the Megatron-LM distributed training framework with the following key design decisions:

1. Deep-and-narrow architecture. Both variants use 32 transformer layers — unusually deep for their parameter count — to better approximate the layer-wise dynamics of billion-scale models, improving proxy fidelity for scaling law extrapolation.
2. WSD learning rate schedule. A Warmup-Stable-Decay schedule was used for stable long-horizon training over 10T tokens.
3. Single tensor-parallel rank. Both models were trained with TP=1 to simplify checkpoint distribution and downstream usage. <br>

Input(s): <br>

Input Type(s): Text <br>

Input Format(s): <br>

• Text: Token IDs (integer sequences) <br>

Input Parameters:

• Text: One-Dimensional (1D) sequence of token IDs <br>

Other Properties Related to Input: These are base (pre-trained) language models. Input is tokenized text. The models accept standard causal-LM input and are not instruction-tuned.

Output(s)

Output Type(s): Text <br>

Output Format(s): <br>

• Text: Next-token logits over vocabulary at each position <br>

Output Parameters:

• Text: Two-Dimensional (2D) — sequence length x vocabulary size <br>

Other Properties Related to Output: As base models, outputs are raw next-token probability distributions. The models are not aligned or instruction-tuned and may produce unfiltered text. <br>

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>

Software Integration:

Runtime Engine:

• Megatron-LM (native checkpoint format) <br>
• Can be converted to HuggingFace Transformers format for inference <br>

Supported Hardware Microarchitecture Compatibility: <br>

• NVIDIA Ampere (A100) <br>
• NVIDIA Hopper (H100, H200) <br>
• NVIDIA Lovelace (L40S) <br>
• CPU inference is feasible given the small model size <br>

Supported Operating System(s):

• Linux <br>

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

Model Version(s):

Both are v1.0 releases. <br>

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality:

• Text <br>

Training Data Size:

Text Training Data Size: 1 trillion tokens <br>

Data Collection Method by dataset:

• Automated <br>

Labeling Method by dataset:

• Not Applicable <br>

Properties: 1 trillion tokens. Content is English-language web text. The data may include publicly available web content of various types (articles, blogs, forums, etc.). <br>

Testing Dataset:

Data Collection Method by dataset:

• Automated <br>

Labeling Method by dataset:

• Not Applicable <br>

Properties: 10 billion tokens. Same source distribution as training data. <br>

Evaluation Dataset:

Data Collection Method by dataset:

• Automated <br>

Labeling Method by dataset:

• Not Applicable <br>

Properties: 10 billion tokens. Same source distribution as training data. <br>

Inference:

Acceleration Engine: Megatron-LM or HuggingFace Transformers (after conversion) <br> Test Hardware: <br>

• NVIDIA A100 / H100 GPU (also runnable on CPU given small size) <br>

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

Bias

Explainability

Safety & Security

Privacy

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>