Resiliency

nemo-mbridge-resiliency

Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.

nvidia/skills·1.9Kinstalls3.5Kstars

SKILL.md

Full skill instructions

Resiliency

Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml

Enablement

Fault tolerance (Slurm only)

Option 1: NeMo Run plugin (recommended)

from megatron.bridge.recipes.run_plugins import FaultTolerancePlugin
import nemo_run as run

task = run.Script(...)
run_plugins = [
    FaultTolerancePlugin(
        enable_ft_package=True,
        calc_ft_timeouts=True,
        num_in_job_restarts=3,
        num_job_retries_on_failure=2,
        initial_rank_heartbeat_timeout=1800,
        rank_heartbeat_timeout=300,
    )
]
run.run(task, plugins=run_plugins, executor=executor)

Plugin parameter	Default	Description
`num_in_job_restarts`	3	Max restarts within same job
`num_job_retries_on_failure`	2	Max new job launches on failure
`initial_rank_heartbeat_timeout`	1800	First heartbeat timeout (seconds)
`rank_heartbeat_timeout`	300	Subsequent heartbeat timeout (seconds)

Option 2: Direct config + ft_launcher

from megatron.bridge.training.config import FaultToleranceConfig

cfg.ft = FaultToleranceConfig(
    enable_ft_package=True,
    calc_ft_timeouts=True,
    simulate_fault=False,
    simulated_fault_type="random",
)

Launch with ft_launcher (not torchrun):

export GROUP_RANK=0  # required for non-Slurm
ft_launcher \
    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \
    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \
    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \
    --ft-rank_out_of_section_timeout=300 \
    your_training_script.py

Config parameter	Default	Description
`enable_ft_package`	False	Enable fault tolerance
`calc_ft_timeouts`	False	Auto-compute optimal timeouts
`simulate_fault`	False	Enable fault simulation for testing
`simulated_fault_type`	`"random"`	`"rank_hung"`, `"rank_killed"`, or `"random"`
`simulated_fault_rank`	None	Specific rank to fault (random if None)
`simulated_fault_base_delay`	0	Base delay before simulating fault

Section-based timeout monitoring covers setup, training steps, checkpointing, and out-of-section time independently. Timeouts are saved to ft_state.json for subsequent runs when calc_ft_timeouts=True.

NVRx straggler detection

from megatron.bridge.training.config import NVRxStragglerDetectionConfig

cfg.nvrx_straggler = NVRxStragglerDetectionConfig(
    enabled=True,
    report_time_interval=300.0,
    calc_relative_gpu_perf=True,
    calc_individual_gpu_perf=True,
    num_gpu_perf_scores_to_print=5,
    gpu_relative_perf_threshold=0.7,
    gpu_individual_perf_threshold=0.7,
    stop_if_detected=False,
    enable_logging=True,
)

Parameter	Default	Description
`enabled`	False	Enable straggler detection
`report_time_interval`	300.0	Seconds between straggler checks
`calc_relative_gpu_perf`	True	Compare ranks against each other
`calc_individual_gpu_perf`	True	Track per-rank degradation over time
`gpu_relative_perf_threshold`	0.7	Threshold for relative performance (0-1)
`gpu_individual_perf_threshold`	0.7	Threshold for individual performance (0-1)
`stop_if_detected`	False	Terminate training on straggler
`num_gpu_perf_scores_to_print`	5	Number of best/worst scores to print
`profiling_interval`	1	Profiling interval for detector

Preemption

Plugin (Slurm)

from megatron.bridge.recipes.run_plugins import PreemptionPlugin

plugins = [
    PreemptionPlugin(
        preempt_time=60,
        enable_exit_handler=True,
        enable_exit_handler_for_data_loader=False,
    )
]

Plugin parameter	Default	Description
`preempt_time`	60	Seconds before job limit to send signal
`enable_exit_handler`	True	Enable signal handler in training
`enable_exit_handler_for_data_loader`	False	Enable for dataloader workers

Direct config

import signal
cfg.train.exit_signal_handler = True
cfg.train.exit_signal = signal.SIGTERM
cfg.train.exit_signal_handler_for_dataloader = False

Re-run state machine (experimental)

from megatron.bridge.training.config import RerunStateMachineConfig

cfg.rerun_state_machine = RerunStateMachineConfig(
    rerun_mode="validate_results",
    check_for_nan_in_loss=True,
    check_for_spiky_loss=False,
    spiky_loss_factor=10.0,
)

Parameter	Default	Description
`rerun_mode`	`"disabled"`	`"disabled"`, `"validate_results"`, `"report_determinism_stats"`
`check_for_nan_in_loss`	True	Check for NaN in loss
`check_for_spiky_loss`	False	Check for unexpectedly large loss
`spiky_loss_factor`	10.0	Loss flagged if > factor * max observed (increase for large models)

Exit codes: 16 = resume to disambiguate, 17 = failed validation.

In-process restart (experimental)

from megatron.bridge.training.config import InProcessRestartConfig

cfg.inprocess_restart = InProcessRestartConfig(
    enabled=True,
    granularity="node",
    soft_timeout=60.0,
    hard_timeout=90.0,
)

Parameter	Default	Description
`enabled`	False	Enable in-process restart
`active_world_size`	None	Ranks executing workload (rest are warm reserves)
`granularity`	`"node"`	`"node"` or `"rank"` restart granularity
`max_iterations`	None	Max restart attempts (None = unlimited)
`soft_timeout`	60.0	Detect GIL-released hangs (seconds)
`hard_timeout`	90.0	Force-terminate hung ranks (seconds)
`heartbeat_interval`	30.0	Heartbeat interval (seconds)
`heartbeat_timeout`	60.0	Missing heartbeat timeout (seconds)
`barrier_timeout`	120.0	Distributed barrier timeout (seconds)
`completion_timeout`	120.0	Completion barrier timeout (seconds)
`empty_cuda_cache`	True	Clear CUDA cache during restart
`max_rank_faults`	None	Max rank faults before terminating
`monitor_process_logdir`	None	Directory for monitor logs

Required environment variables:

export TORCH_CPP_LOG_LEVEL=error
export TORCH_NCCL_RETHROW_CUDA_ERRORS=0
export NCCL_NVLS_ENABLE=0

The PyTorch NCCL watchdog timeout must exceed hard_timeout. NeMo-Run's Slurm Executor is not supported; launch directly with srun --kill-on-bad-exit=0.

Async checkpoint save

cfg.checkpoint.async_save = True
cfg.checkpoint.ckpt_format = "torch_dist"

Local checkpointing (NVRx)

cfg.checkpoint.non_persistent_local_ckpt_dir = "/local/scratch/ckpt"
cfg.checkpoint.non_persistent_local_ckpt_algo = "fully_parallel"

Code Anchors

Fault tolerance

Config: src/megatron/bridge/training/config.py — FaultToleranceConfig
Runtime: src/megatron/bridge/training/fault_tolerance.py
Plugin: src/megatron/bridge/recipes/run_plugins.py — FaultTolerancePlugin
Perf plugin: scripts/performance/nemo-mbridge-resiliency_plugins.py
Tests: tests/unit_tests/training/test_fault_tolerance.py
Example: examples/training_features/nemo-mbridge-resiliency/fault_tolerance/

Straggler detection

Config: src/megatron/bridge/training/config.py — NVRxStragglerDetectionConfig
Runtime: src/megatron/bridge/training/nvrx_straggler.py
Train loop: src/megatron/bridge/training/train.py — check_nvrx_straggler_detection
Tests: tests/unit_tests/training/test_nvrx_straggler.py, tests/functional_tests/training/test_nvrx_straggler.py
Example: examples/training_features/nemo-mbridge-resiliency/straggler_detection/

In-process restart

Config: src/megatron/bridge/training/config.py — InProcessRestartConfig
Runtime: src/megatron/bridge/training/inprocess_restart.py
Entry point: src/megatron/bridge/training/pretrain.py — maybe_wrap_for_inprocess_restart
Tests: tests/unit_tests/training/test_inprocess_restart.py, tests/functional_tests/training/test_inprocess_restart.py

Preemption

Plugin: src/megatron/bridge/recipes/run_plugins.py — PreemptionPlugin
Signal handler: src/megatron/bridge/training/utils/sig_utils.py
Tests: tests/unit_tests/recipes/test_run_plugins.py

Re-run state machine

Config: src/megatron/bridge/training/config.py — RerunStateMachineConfig
Init: src/megatron/bridge/training/initialize.py — init_rerun_state

Checkpointing

Async save: src/megatron/bridge/training/checkpointing.py — schedule_async_save
Local ckpt: src/megatron/bridge/training/checkpointing.py — LocalCheckpointManager
Tests: tests/functional_tests/training/test_local_checkpointing.py

Pitfalls

ft_launcher, not torchrun: Direct FaultToleranceConfig requires ft_launcher. Using torchrun silently disables FT. For non-Slurm, set GROUP_RANK=0.
Async save requires torch_dist: async_save=True only works with ckpt_format="torch_dist". Other formats silently fail or error.
IPR + NeMo-Run: In-process restart is not compatible with NeMo-Run or Slurm preemption plugins. Requires specific PyTorch/NCCL versions and env vars.
NVRx vs legacy straggler: Two detectors exist. Use NVRx (nvrx_straggler); do not enable both.
stop_if_detected default: NVRx logs but does not stop training by default. Set stop_if_detected=True for automatic termination.
NCCL watchdog vs hard_timeout: For IPR, NCCL watchdog timeout must exceed hard_timeout or PyTorch kills the process before recovery.
Rerun state machine is alpha: Use check_for_nan_in_loss=True for NaN detection, but don't rely on full rerun workflows yet.

Verification

Fault tolerance

./examples/training_features/nemo-mbridge-resiliency/fault_tolerance/run_fault_tolerance.sh
./examples/training_features/nemo-mbridge-resiliency/fault_tolerance/run_fault_tolerance.sh --simulate-fault

Look for [FaultTolerance] / [RankMonitorServer] log lines with section timeouts. Simulated fault should trigger restart from checkpoint.

Straggler detection

uv run python -m torch.distributed.run --nproc_per_node=2 \
    examples/training_features/nemo-mbridge-resiliency/straggler_detection/straggler_detection_example.py

Look for GPU relative performance and GPU individual performance reports with per-rank scores.

Async checkpoint

Look for Scheduling async checkpoint save in logs. Training iterations should continue while checkpoint files are being written.

In-process restart

pytest tests/functional_tests/training/test_inprocess_restart.py -v

Requires compatible PyTorch/NCCL versions.

Install skill

Copy and run in your terminal

npx -y skills add https://github.com/nvidia/skills --skill nemo-mbridge-resiliency

View on GitHub

nemo-mbridge-resiliency

rag-blueprint
NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage. Handles any RAG action: deploy, install, start, enable, disable, toggle, change, configure, troubleshoot, debug, fix, shutdown, stop, or tear down any RAG feature or service (Agentic RAG, VLM, guardrails, query rewriting, models, search, ingestion, observability, summarization, reasoning, and more).
3.1K installs
accelerated-computing-cudf
Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable semantics, and multi-GPU DataFrame workloads.
3K installs
cuopt-developer
Modify, build, test, debug, and contribute to NVIDIA cuOpt (C++/CUDA, Python, server, CI). Use for solver internals, PRs, DCO, and code conventions.
2.7K installs
cuopt-install
Install cuOpt for Python, C, or server via pip, conda, or Docker; verify the install. For building cuOpt from source, see cuopt-developer.
2.5K installs

More skills from nvidia

nvidia/skills

cudaq-guide

CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.

3.5K 2.5K

View

nvidia/skills

data-designer

Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline.

3.5K 2.4K

View

nvidia/skills

deepstream-dev

NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.

3.5K 2.4K

View

nvidia/skills

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

3.5K 2.3K

View

nvidia/skills

nemo-retriever

Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (`.jpg` `.png` `.tiff`), Office (`.docx` `.pptx`), text (`.html` `.txt`), audio (`.mp3` `.wav` `.m4a`), or video (`.mp4` `.mov`). Prefer this over native Read / Grep for multi-file or non-PDF corpora. Not for: editing files, web browsing, single-file plain-text lookups, fine-tuning.

3.5K 2.3K

View

nvidia/skills

omniverse-cad-to-simready

Coordinate the end-to-end CAD/source-asset to SimReady workflow. Use for broad requests such as CAD to SimReady, source asset to simulation-ready USD, or prop packaging that require conversion, material/physics assignment, SimReady conformance, validation, and optional package creation; deploy or verify Content Agents services first when property assignment is enabled; route single-stage work through nested references.

3.5K 2.3K

View

nvidia/skills

deepstream-import-vision-model

Use this skill to bring any vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report. Object detection models only.

3.5K 2.3K

View

nvidia/skills

omniverse-usd-performance-tuning

Top-level workflow skill for USD performance diagnosis and optimization. Handles slow loading, high memory, low FPS, and broad scene-optimization requests; delegates auth/runtime setup to Phase 0 owners.

3.5K 2.3K

View

nvidia/skills

cuopt-routing-api-python

Vehicle routing (VRP, TSP, PDP) with cuOpt — Python API only. Use when the user is building or solving routing in Python.

3.5K 2.2K

View

nvidia/skills

skill-card-generator

Use only to generate or update a governance skill card for a specified existing agent skill directory. Do not use for explaining, listing, comparing, or discussing skill capabilities.

3.5K 2.2K

View

nvidia/skills

omniverse-realtime-viewer

Use as the top-level router for Omniverse Realtime Viewer USD app requests and focused viewer reference documents.

3.5K 2.2K

View

nvidia/skills

rag-perf

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

3.5K 2.1K

View