GR00T-N1.7-G1-BrainCo-Pick

GR00T N1.7 — Unitree G1 BrainCo Pick (Ego-Centric)

Fine-tuned checkpoint of nvidia/GR00T-N1.7-3B on 1,598 real-robot episodes of tabletop pick-and-place tasks performed by a Unitree G1 humanoid equipped with BrainCo dexterous hands.

This model uses a single ego-centric head camera (no wrist cameras) and predicts 26-DOF arm + hand joint commands for left-arm manipulation.

---

Model Details

---

Training Data

8 picking tasks, 1,598 total episodes from the Unitree G1 BrainCo dataset:

Data format: GR00T-flavored LeRobot v2.1 (per-episode HDF5-backed parquet + mp4 videos).

Data pipeline: Raw episodes were captured as human-teleoperated demonstrations, IK-retargeted to the G1 robot via training_data_g1brainco_newik.npz, and converted from LeRobot v3.0 to v2.1 format with the preprocessing script at examples/G1_Brainco/preprocess_brainco_pick.py in the Isaac GR00T repo.

---

Training Configuration

base_model:              nvidia/GR00T-N1.7-3B
embodiment_tag:          NEW_EMBODIMENT
max_steps:               20000
learning_rate:           1e-4
warmup_ratio:            0.05
weight_decay:            1e-5
optimizer:               paged_adamw_8bit
global_batch_size:       1
gradient_accumulation:   64          # effective batch = 64
gradient_checkpointing:  true
num_shards_per_epoch:    2000
shard_size:              512
episode_sampling_rate:   0.1
color_jitter:            brightness=0.3, contrast=0.4, saturation=0.5, hue=0.08
tune_llm:                false
tune_visual:             false
tune_projector:          true
tune_diffusion_model:    true

Loss Curve

---

Modality Configuration

# g1_brainco_config_pick_ego.py
config = {
    "video": ModalityConfig(
        delta_indices=[0],
        modality_keys=["observation.images.head_stereo_left"],  # ego-centric only
    ),
    "state": ModalityConfig(
        delta_indices=[0],
        modality_keys=[
            "observation.state.arm_q",       # 14D arm joint positions (rad)
            "observation.state.hand_state",  # 12D BrainCo finger state
        ],
    ),
    "action": ModalityConfig(
        delta_indices=list(range(16)),       # 16-step prediction horizon
        modality_keys=[
            "action.arm_q",    # 14D relative arm joint targets
            "action.hand_cmd", # 12D absolute hand commands
        ],
        action_configs=[
            ActionConfig(rep=ActionRepresentation.RELATIVE, type=ActionType.NON_EEF,
                         state_key="observation.state.arm_q"),
            ActionConfig(rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF),
        ],
    ),
    "language": ModalityConfig(
        delta_indices=[0],
        modality_keys=["annotation.human.task_description"],
    ),
}

---

Quick Start

Install

git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
uv sync --all-extras
# or: pip install -e ".[dev]"

Download the checkpoint

from huggingface_hub import snapshot_download

model_path = snapshot_download("JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick")

Run inference server

python gr00t/eval/run_gr00t_server.py \
    --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path modality_config.py \
    --denoising-steps 4

Open-loop evaluation

python gr00t/eval/open_loop_eval.py \
    --dataset-path /path/to/brainco_pick/G1_Brainco_PickApple_Dataset \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \
    --modality-config-path modality_config.py

Python inference

import numpy as np
from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline

pipeline = Gr00tN1d7Pipeline.from_pretrained(
    "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick",
    modality_config_path="modality_config.py",
)

# Single inference call
output = pipeline.get_action(
    observation={
        # Head camera frame: (H, W, 3) uint8
        "observation.images.head_stereo_left": head_frame,
        # 14D arm joint positions (radians)
        "observation.state.arm_q": arm_q,
        # 12D BrainCo finger state
        "observation.state.hand_state": hand_state,
    },
    language_instruction="Pick up the apple and place it in the bowl",
)

# output["action.arm_q"]   → (16, 14) relative arm delta targets
# output["action.hand_cmd"] → (16, 12) absolute hand commands

---

Deploying on Real Unitree G1 + BrainCo Hands

Hardware requirements

• Unitree G1 humanoid robot
• BrainCo dexterous hands (6-DOF per hand)
• Head stereo-left camera (640×480 RGB, 30 fps)
• Inference host: ≥16 GB VRAM GPU (RTX 4090 / H100 / Jetson Thor)

Action space mapping

output["action.arm_q"]    (16, 14) — relative delta from current arm_q
    [0:7]   left arm  : shoulder_pitch, shoulder_roll, shoulder_yaw,
                         elbow_pitch, elbow_roll, wrist_pitch, wrist_yaw
    [7:14]  right arm : (same order; right arm stationary in training data)

output["action.hand_cmd"] (16, 12) — absolute finger positions [0, 1]
    [0:6]   left hand : thumb, index, middle, ring, pinky + spread
    [6:12]  right hand: (same order)

Control loop (30 Hz skeleton)

import numpy as np
from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline

pipeline = Gr00tN1d7Pipeline.from_pretrained(
    "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick",
    modality_config_path="modality_config.py",
)

action_buffer = None
action_step = 0
HORIZON = 16

while True:
    # 1. Read sensors
    head_img  = robot.get_camera("head_stereo_left")          # (480, 640, 3) uint8
    arm_q     = robot.get_arm_joint_positions()               # (14,) radians
    hand_state = robot.get_hand_state()                       # (12,) [0,1]

    # 2. Infer (every HORIZON steps to amortise diffusion cost)
    if action_step == 0 or action_buffer is None:
        output = pipeline.get_action(
            observation={
                "observation.images.head_stereo_left": head_img,
                "observation.state.arm_q": arm_q,
                "observation.state.hand_state": hand_state,
            },
            language_instruction="Pick up the apple",
        )
        action_buffer = output
        action_step = 0

    # 3. Execute current step
    delta_arm  = action_buffer["action.arm_q"][action_step]    # (14,)
    hand_cmd   = action_buffer["action.hand_cmd"][action_step] # (12,)

    target_arm = arm_q + delta_arm
    target_arm = np.clip(target_arm, -np.pi, np.pi)

    robot.set_arm_joints(target_arm, kp=15, kd=0.5)
    robot.set_hand_joints(hand_cmd)

    action_step = (action_step + 1) % HORIZON
    time.sleep(1 / 30.0)

---

Files in this Repository

---

Reproducing Training

1. Clone the repo

   git clone https://github.com/NVIDIA/Isaac-GR00T
   cd Isaac-GR00T
   uv sync --all-extras
   

2. Download and preprocess the datasets

   python examples/G1_Brainco/preprocess_brainco_pick.py
   # Downloads 8 datasets from unitreerobotics HF org → dataset_g1/brainco_pick/
   

3. Launch fine-tuning

   bash examples/G1_Brainco/finetune_g1_brainco_pick.sh
   

   Hardware notes for 32 GB VRAM (RTX 5090):

   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
   --gradient_checkpointing
   --optim paged_adamw_8bit
   --global_batch_size 1 --gradient_accumulation_steps 64
   # Peak ~26 GB, ~5.8 s/step, 20 000 steps ≈ 32 h
   

---

Limitations

• Trained on left-arm only demonstrations; right-arm commands are zero throughout.
• Single ego-centric head camera — no wrist-camera feedback. Fine grasping of very small objects may be less precise than a 3-camera setup.
• Data collected in a specific lab tabletop environment; real-world lighting and clutter variation may affect performance.
• Not trained for whole-body or locomotion control.

---

Citation

If you use this checkpoint, please cite the base GR00T N1.7 model:

@misc{gr00tn17_2025,
  title  = {Isaac GR00T N1.7: Open Foundation Model for Generalized Humanoid Control},
  author = {NVIDIA},
  year   = {2025},
  url    = {https://huggingface.co/nvidia/GR00T-N1.7-3B}
}

And the Unitree G1 BrainCo datasets:

@misc{unitree_brainco_2025,
  title  = {Unitree G1 BrainCo Manipulation Dataset},
  author = {Unitree Robotics},
  year   = {2025},
  url    = {https://huggingface.co/unitreerobotics}
}

---

License

Weights are released under the same Apache 2.0 license as the base GR00T N1.7 model.