gr00t-nth-lr-vpconcat-26k

GR00T N1.6 PolaRiS NTH-LR VPConcat — checkpoint 26K

This is the complete step-26,000 checkpoint from a selective fine-tune of nvidia/GR00T-N1.6-3B on ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot. It consumes two RGB views, Franka joint/gripper state, and an English instruction and predicts a 16-step joint/gripper action chunk.

This model is not a LoRA adapter. It uses NVIDIA Isaac-GR00T N1.6's standard NEW_EMBODIMENT selective fine-tuning path and contains full model weight shards.

This is an intermediate snapshot at 26,000 of 88,000 planned optimizer updates. It is not the final selected policy. Closed-loop MolmoSpaces results have not yet been recorded for this checkpoint.

At a glance

Intended use and limitations

This checkpoint is intended for research/evaluation of language-conditioned Franka manipulation with exactly the schema below. It supports direct Gr00tPolicy inference, the GR00T policy server, open-loop dataset inspection, and exact Trainer resume when training state is retained.

It is not a generic Hugging Face text/image pipeline, so hosted inference is disabled. It is not an end-effector-delta policy and must not be connected to a robot or simulator with a different joint order, gripper convention, camera convention, frequency, normalization, or execution semantics. Use strict input validation. No closed-loop success rate is reported; training loss is not a benchmark result. The inherited NVIDIA license restricts use to non-commercial research or evaluation. Read LICENSE.

Download

The full repo is about 22.8 GB: roughly 9.8 GB of model weights plus a roughly 12.9 GB AdamW optimizer and small resume/config files. For inference only:

hf download ccwatson/gr00t-nth-lr-vpconcat-26k \
  --local-dir ./gr00t-nth-lr-vpconcat-26k \
  --exclude optimizer.pt \
  --exclude scheduler.pt \
  --exclude rng_state.pth \
  --exclude trainer_state.json \
  --exclude training_args.bin

For exact resume or archival inspection, download everything:

hf download ccwatson/gr00t-nth-lr-vpconcat-26k \
  --local-dir ./gr00t-nth-lr-vpconcat-26k

Validated software version

git clone --branch n1.6.1-release https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
git checkout 5dc80c4afd726b34faad1d8f7e007a13b34e4c88
uv sync --frozen

config.json uses Gr00tN1d6; the Isaac-GR00T package registers that Transformers class. Newer revisions may work but were not used for training or qualification.

Direct inference

The saved processor_config.json contains the complete new_embodiment contract and normalization. Re-registering the training-time Python config is not required merely to load this checkpoint.

import numpy as np
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.policy.gr00t_policy import Gr00tPolicy

policy = Gr00tPolicy(
    model_path="./gr00t-nth-lr-vpconcat-26k",
    embodiment_tag=EmbodimentTag.NEW_EMBODIMENT,
    device="cuda:0",
    strict=True,
)

# exterior_rgb/wrist_rgb: RGB uint8 (H,W,3); q: 7 joints; gripper: scalar.
observation = {
    "video": {
        "exterior_1_left": exterior_rgb[None, None, ...],  # (1,1,H,W,3)
        "wrist_left": wrist_rgb[None, None, ...],
    },
    "state": {
        "joint_position": np.asarray(q, np.float32)[None, None, :],  # (1,1,7)
        "gripper_position": np.asarray([gripper], np.float32)[None, None, :],
    },
    "language": {
        "annotation.language.language_instruction": [[instruction]],
    },
}
action, info = policy.get_action(observation)
assert action["joint_position"].shape == (1, 16, 7)
assert action["gripper_position"].shape == (1, 16, 1)
next_joint_target = action["joint_position"][0, 0]
next_gripper_target = action["gripper_position"][0, 0]

training_recipe/inference_example.py is a runnable version accepting two image files, seven joint values, one gripper value, and an instruction.

Observation contract

Every input has delta_indices=[0], hence temporal horizon 1. Images must be RGB, not BGR. Joint order, units, and gripper convention must match the dataset.

Action contract

Training data is 15 FPS and the prediction horizon is 16. Prediction horizon and executed horizon are different: validate how many steps to execute before replanning in the target environment. Do not reinterpret arm output as Cartesian deltas.

The root model config retains padded action_horizon=50 and 128-wide tensors. Those are cross-embodiment padding maxima. The saved new_embodiment modality config is authoritative here: 16 steps, 7 arm values, and 1 gripper value.

Policy server

uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /absolute/path/to/gr00t-nth-lr-vpconcat-26k \
  --embodiment-tag NEW_EMBODIMENT \
  --device cuda:0 \
  --strict \
  --host 0.0.0.0 \
  --port 5555

Clients send the nested observation above. The server owns the checkpoint and returns a 16-step chunk; the client owns the execution horizon and replanning. There is no published MolmoSpaces/vla-evaluation-harness config yet, so this is the appropriate validated GR00T-native configuration, not a claim that closed-loop integration is complete.

Open-loop dataset check

Register the custom config before open_loop_eval.py parses its CLI:

PYTHONPATH="/path/to/model/training_recipe:${PYTHONPATH:-}" \
uv run python -c '
import runpy
import pnpnext_lr_jointpos_config
runpy.run_module("gr00t.eval.open_loop_eval", run_name="__main__")
' \
  --dataset-path /path/to/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /path/to/gr00t-nth-lr-vpconcat-26k \
  --traj-ids 0 \
  --action-horizon 16 \
  --save-plot-path /tmp/gr00t-26k-open-loop.jpeg

This is prediction-vs-ground-truth diagnostics, not closed-loop success.

Dataset and modality configuration

All 3,940 episodes from this exact dataset revision were eligible, matching the corresponding OpenPI pi0.5 data selection:

hf download \
  ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --repo-type dataset \
  --revision c74ec0af9b73a8ceca5af70e028d247ace6b8d0f \
  --local-dir /path/to/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot

For GR00T we explicitly selected seven arm joints and one gripper dimension; we did not assume OpenPI and GR00T share a schema. Normalization is included in statistics.json and experiment_cfg/dataset_statistics.json.

The exact registration file is training_recipe/pnpnext_lr_jointpos_config.py. Its contract is:

PNPNEXT_LR_JOINTPOS_CONFIG = {
    "video": ModalityConfig([0], ["exterior_1_left", "wrist_left"]),
    "state": ModalityConfig([0], ["joint_position", "gripper_position"]),
    "action": ModalityConfig(
        list(range(16)), ["joint_position", "gripper_position"],
        action_configs=[
            ActionConfig(rep=RELATIVE, type=NON_EEF, format=DEFAULT,
                         state_key="joint_position"),
            ActionConfig(rep=ABSOLUTE, type=NON_EEF, format=DEFAULT),
        ],
    ),
    "language": ModalityConfig(
        [0], ["annotation.language.language_instruction"]
    ),
}

Exact training command

At stack revision a5d3a1d, the worker ran the following. /tmp inputs were checksummed copies staged from persistent Kostas GRAID onto node-local NVMe; checkpoint output remained on persistent GRAID.

uv run --frozen --no-sync python gr00t/experiment/launch_finetune.py \
  --base-model-path /tmp/isaac-gr00t-520010/base-model \
  --dataset-path /tmp/isaac-gr00t-520010/lerobot/ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path /mnt/kostas-graid/datasets/ccwatson/repos/molmo-robotics-stack/scripts/isaac_gr00t/pnpnext_lr_jointpos_config.py \
  --num-gpus 1 \
  --output-dir /mnt/kostas-graid/datasets/ccwatson/molmo-robotics-stack-state/isaac-gr00t/checkpoints/pnpnext-lr-jointpos \
  --save-total-limit 5 \
  --save-steps 500 \
  --max-steps 88000 \
  --global-batch-size 32 \
  --color-jitter-params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08 \
  --dataloader-num-workers 4 \
  --experiment-name gr00t-n1d6-pnpnext-lr-jointpos-grasp-l40-20260824 \
  --use-wandb

training_recipe/train_command.sh is the same command with portable path environment variables.

Resolved training configuration

NVIDIA defaults not explicit above are captured in the checkpoint's fully resolved experiment_cfg/conf.yaml.

Trainable modules: top four language-model layers, projector, diffusion model, and VLLN. Frozen: visual backbone and remaining language-model layers. Flags: tune_top_llm_layers=4, tune_llm=false, tune_visual=false, tune_projector=true, tune_diffusion_model=true, tune_vlln=true.

Batch-size and duration qualification

Batch 32 left 5,402 MiB and met a 4 GiB reserve. A 100-update smoke measured 0.929293 seconds/steady update and 64.7214 seconds/full checkpoint write. Including saves and 12-hour requeues, 88,000 updates projected to 29.97 hours. See training_recipe/batch_probe.csv.

Checkpointing, resume, and validation

GRASP Slurm job 520010 used 12-hour allocations, --requeue, and --signal=B:USR1@300. On signal the worker stops Trainer, requeues the same job, restages inputs locally, and relaunches. Isaac-GR00T uses resume_from_checkpoint=True, selecting the newest checkpoint in the same experiment directory. Job 519994 qualified the path by writing step 50, requeueing under the same ID, and continuing through step 100.

For exact step-26K resume, place this directory at <output-dir>/<experiment-name>/checkpoint-26000, keep optimizer/scheduler/RNG and Trainer files, and relaunch the same command with the same source, dataset, config, arguments, and experiment name. An inference-only download is not resumable.

Qualification also covered dataset/statistics and relative-action round trips, batch-32/100-step smokes, exact loading with no key mismatch, real two-camera inference with finite (1,16,7) and (1,16,1) outputs, and finite nonzero CUDA gradients through action decoding and differentiable Franka FK.

FK calibration used 4,096 recorded joint/TCP pairs. Asset constants are 580 mm pedestal-to-base and 262 mm link-7-to-grasp-site. Fitted offsets were 579.978 mm and 261.977 mm; median/p99 position error was 0.133/0.383 mm and p99 orientation error 0.052°. See training_recipe/fk_calibration.json. These checks do not establish benchmark performance or robot safety.

Provenance

Structured metadata: training_recipe/provenance.json.

File guide

Evaluation status

Closed-loop MolmoSpaces evaluation is pending. Future results should identify the exact Hub/checkpoint revision, benchmark leaf, renderer, execution horizon, episode coverage, W&B eval run, and artifact root. Do not infer success from training loss or the open-loop smoke.