Nemotron-3-Labs-Ultra-Math-SFT

Nemotron-3-Labs-Ultra-Math-SFT

Referred to as Nemotron-3-Ultra-SFT in the technical report.

Description

Nemotron-3-Labs-Ultra-Math-SFT is a decoder-only transformer language model specialized for mathematical reasoning, trained to solve difficult mathematical problems and identify mistakes in proofs, and deployed as part of an ensemble system that achieved a gold-medal level score at the International Mathematical Olympiad 2026.

Full details can be found at our technical report An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics.

Nemotron-3-Labs-Ultra-Math-SFT was developed by NVIDIA as a part of Nemotron.

This model is ready for commercial and non-commercial use.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1).

Deployment Geography

Global

Use Case

Researchers and developers focused on AI-driven mathematical reasoning and proof verification, aiming to advance open models for solving complex math problems and improving reasoning capabilities.

Release Date

HuggingFace: September 3, 2026 via https://huggingface.co/collections/nvidia/nemotron-labs-imo-2026

Reference(s)

• An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics (technical report)
• nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
• nvidia/Nemotron-Math-Proofs-v3-SFT
• nvidia/Nemotron-IMO-Bench
• Inference pipeline and submitted proofs: https://github.com/NVIDIA-NeMo/Skills/tree/main/recipes/nemotron-imo-tts
• RL training recipe: https://github.com/NVIDIA-NeMo/RL/blob/imo-26-ultra-v3/docs/guides/nemotron-3-ultra-imo.md

Model Architecture

Architecture Type: Mamba2-Transformer Hybrid Latent Mixture of Experts (LatentMoE) with Multi-Token Prediction (MTP) Network Architecture: Nemotron Hybrid LatentMoE This model was developed based on nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16. Number of model parameters: 550B Total / 55B Active

Input

Input Type(s): Text Input Format(s): String Input Parameters: One-Dimensional (1D) Other Properties Related to Input: Maximum context length up to 1M tokens

Output

Output Type(s): Text Output Format: String Output Parameters: One-Dimensional (1D) Other Properties Related to Output: Maximum context length up to 1M tokens

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s): vLLM Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper

Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Nemotron-3-Labs-Ultra-Math-SFT v1

Deployment

The Ultra BF16 checkpoint is a frontier-scale model. The minimum recommended hardware is:

• Single-node: 8× B200 (≈1.5 TB aggregate HBM — fits BF16 weights plus KV cache with headroom)
• Multi-node: ≥8 GPUs across H100 / H200 / GB200 / GB300, orchestrated with Ray v2

For more detailed information, please see the nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 model card and this cookbook.

Recommended container: vllm/vllm-openai:v0.22.0.

export MODEL_CKPT=PATH/TO/MODEL/CHECKPOINT

8× B200 single-node deployment:

docker run -d --name nemotron-ultra-vllm \
  --gpus all \
  --ipc=host \
  --network=host \
  --shm-size=16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v $MODEL_CKPT:/model:ro \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e SAFETENSORS_FAST_GPU=1 \
  -e NVIDIA_TF32_OVERRIDE=1 \
  -e VLLM_LOGGING_LEVEL=INFO \
  vllm/vllm-openai:v0.22.0 \
  /model \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name nvidia/Nemotron-3-Labs-Ultra-Math-SFT \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mamba-ssm-cache-dtype float16 \
  --mamba-backend flashinfer \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}'

Context length defaults to 256k above. To use up to 1M, set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and --max-model-len 1048576.

Training, Testing, and Evaluation Datasets

Training Dataset

Data Modality: Text

Text Training Data Size: 39,047,191,715 recorded tokens (414,890 samples) Data Collection Method by dataset: Hybrid: Automated, manually-collected, Synthetic Labeling Method by dataset: Hybrid: Automated, manually-labeled, Synthetic Properties (Quantity, Dataset Descriptions, Sensor(s)): Nemotron-Math-Proofs-v3-SFT is a long-form mathematical reasoning dataset containing proof-generation, proof-refinement, verification, and meta-verification traces. The release contains 414,890 samples representing 15,818 unique problems after quality filtering. The source pool contains 15,879 hard proof problems selected from the AoPS subset of nvidia/Nemotron-Math-Proofs-v1. Responses are generated using DeepSeek-V4-Pro in Max inference mode. A four-round generate-verify-refine pipeline produces initial proofs in round 1 and refinement trajectories for problems that remain unsolved in rounds 2 through 4. The pipeline also produces verifier and meta-verifier traces following the proof-generation and proof-verification prompting style described in the DeepSeekMath-V2 paper (Shao et al., 2025). See nvidia/Nemotron-Math-Proofs-v3-SFT.

Testing Dataset

Data Collection Method by dataset: Hybrid: Automated, manually-collected, Synthetic<br> Labeling Method by dataset: Hybrid: Automated, manually-labeled, Synthetic<br> Properties (Quantity, Dataset Descriptions, Sensor(s)): This corpus comprises only benchmarks for assessing the quality of mathematical proof generation and verification.

Evaluation Dataset

Benchmark Score: As part of an ensemble system, this model achieved a gold-medal-level score at the International Mathematical Olympiad 2026. Additional evaluation results are available in the Accompanying Tech Report.

Data Collection Method by dataset: Hybrid: Automated, manually-collected, Synthetic<br> Labeling Method by dataset: Hybrid: Automated, manually-labeled, Synthetic<br> Properties (Quantity, Dataset Descriptions, Sensor(s)): This corpus comprises only benchmarks for assessing the quality of mathematical proof generation and verification.

Inference

Acceleration Engine: vLLM, PyTorch Hardware Requirements (GPU Architecture, Model):

• NVIDIA Ampere - A100 80 GB
• NVIDIA Hopper - H100

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.