gr00t-n17-g1-table-rel-60k

GR00T N1.7 — Unitree G1 bimanual table assembly (RELATIVE end-effector, 60k)

Fine-tune of nvidia/GR00T-N1.7-3B for a bimanual children's-table assembly task on a Unitree G1 with Dex1-1 grippers. Inference-only bundle (optimizer states excluded).

• Dataset: BitRobot/G1_WBT_Dex1_Building-Children-Table — 533 episodes, LeRobot v3.0, 30 fps, converted here to a GR00T-flavoured v2.1 set at 10 fps (2,092,332 frames), split 479 train / 54 val
• Checkpoint: step 60,000 (final; the schedule is cosine so this is the intended end point)
• Training: 4×A100-80GB, 60.3 h, effective batch 256 (per-device 64 × 4 GPUs × accum 1), lr 1e-4 cosine, warmup 5 %, weight decay 1e-5, bf16 + tf32, DeepSpeed ZeRO-2, 70.8 samples/s. VLM backbone (Cosmos-Reason2-2B) frozen — only projector, diffusion head, linear and vlln layers are trained. Train loss 1.123 → 0.0137.

Action / observation space

End-effector values are body-frame (x forward, y left, z up; metres for position, radians for roll/pitch/yaw), gripper aperture in Dex1 units clamped to [0, 4.5]. Arms use a RELATIVE representation — the policy predicts deltas internally, and the server reconstructs absolute targets against the current state before returning them, so clients always receive absolute poses. The gripper is ABSOLUTE.

⚠️ The origin link of the body frame (pelvis vs torso) is not documented upstream and was not independently confirmed; axis directions were verified empirically from the data. Confirm the frame convention against your controller before deployment.

⚠️ This policy is subtask-conditioned, not a single-instruction policy

The dataset carries a per-frame subtask label, and the model learned to condition on it. The label set is:

insert table leg to table base · move to table · flip table · rotate leg to tighten · pick table leg · rotate table base · building children table · move table base

building children table is not a global instruction covering the whole episode — measured over 80 episodes it appears only in the opening 1–6 s of each one, and the rest of the episode cycles through [move to table →] pick → insert → rotate-leg-to-tighten → rotate-table-base, once per leg. 98.8 % of frames therefore carry a specific subtask label.

Consequence for deployment: feeding a single overall instruction puts the policy far outside its training conditioning. You need an instruction scheduler (a state machine or a higher-level planner) that switches the string as the task progresses, using the exact label strings above.

Execution

Returned chunks are 16 steps at 10 fps. Executing roughly half a chunk (8 steps, 0.8 s) before re-observing is a reasonable default; a shorter execution horizon in contact-rich phases and a longer one in free space is the natural refinement. With an end-to-end latency of ~160 ms the constraint execution_steps × 100 ms > latency is satisfied from 2 steps upward, so the horizon can be tuned freely.

python gr00t/eval/run_gr00t_server.py \
  --model-path <this repo> --embodiment-tag new_embodiment --port 5555

Validated against Isaac-GR00T at commit 65cc4a192, transformers 4.57.3, torch 2.7.1+cu128. processor_config.json and statistics.json in this repo carry the modality config and normalization statistics, so no external config import is needed to serve the model. Reading the gated nvidia/Cosmos-Reason2-2B tokenizer (or a pre-populated offline cache) is required.

Because the vision-language backbone is frozen, deployed camera placement, field of view and exposure should match the training distribution; large visual shifts degrade performance sharply. This matters particularly for real→sim transfer, where rendered imagery differs from the recorded RGB the model was trained on.

Evaluation

Open-loop evaluation on the held-out 54 episodes is running and the numbers will be added here. For reference, an earlier run of the same recipe on different hardware (2×B200, per-device 16 × accum 8, same effective batch 256) reached val54 MSE 0.0409 / MAE 0.0829 at 60k, with per-subtask MSE ranging from 0.0189 (rotate leg to tighten, the most frequent subtask) to 0.0757 (building children table, the rarest). The weakest subtasks are consistently the least frequent ones, so additional data on those segments is the main expected lever.

Open-loop error is a proxy. Closed-loop evaluation on the robot is the real test, and open-loop metrics should not be used on their own to rank candidates.