gr00t-n1.7-so101-molmoact2

GR00T N1.7: SO-101 fine-tune on MolmoAct2 community data

LoRA fine-tune of nvidia/GR00T-N1.7-3B (VLM backbone frozen; adapters on the action head) for the SO-101 follower arm, trained on the SO-101 subset of allenai/MolmoAct2-SO100_101-Dataset with MolmoAct2's annotated language instructions. Adapters (55.4M params, 1.73% of 3.2B; via peft + category-lora) are merged into this repo's bf16 weights; raw adapters are in adapters/.

Intended use & safety

• Intended use: research and evaluation on SO-101 follower arms (single 6-dof arm, front + wrist cameras) for the tabletop manipulation task families present in the data (pick/place, stacking, sorting, and similar).
• Out of scope: any other embodiment or rig without fine-tuning (VLA policies do not zero-shot transfer across embodiments); unattended operation; operation near people without a hardware e-stop and enforced workspace/torque limits.
• Validation status: offline held-out action loss only (see Losses & evaluation); no sim or real-robot rollouts yet. Users are responsible for safe integration (guardrails, e-stop, workspace limits) before any deployment.

Training

LR sweep (250 steps, bs=64, same eval): 5e-5→0.97, 1e-4→0.79, 3e-4→0.59, 6e-4→0.44, 1.2e-3→0.79. 6e-4 won short-horizon but NaN'd the full run at step ~2.5k; 3e-4 trained clean end-to-end.

Losses & evaluation

• Training loss: GR00T N1.7's flow-matching action objective. The DiT action head predicts the velocity field for a noised 16-step action chunk (conditioned on frozen VLM features + robot state), MSE against the interpolation target. Trainable: LoRA adapters on 252 action-head Linears + 7 CategorySpecificLinear, plus action-head norms. Frozen: LLM, vision encoder, all base weights.
• Eval regime: offline action loss. The same flow-matching objective on the 112 held-out episodes (episode-level split, 5%/repo, seed 17). 384 cached samples drawn round-robin from all 39 repos' test splits, processed by the training processor instance (identical normalization statistics); flow-matching noise/timesteps seeded via fork_rng so values are comparable across steps and runs. Upstream Isaac-GR00T has no working eval path; ours is implemented in the training repo (src/callbacks.py).
• Scope: this is an action-prediction loss on same-task/unseen-trajectory data; it is not a task success rate and does not measure cross-task or cross-scene generalization.

Provenance

Usage

from gr00t.model.policy import Gr00tPolicy
policy = Gr00tPolicy.from_pretrained("robocurve/gr00t-n1.7-so101-molmoact2")

• Observations: cameras front (third-person) and wrist (gripper-mounted), letterboxed to 256×256; state single_arm → [0:5], gripper → [5:6] in LeRobot so101_follower joint conventions (shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper).
• Actions: same 6-dof layout; single_arm uses GR00T's relative-action representation, gripper absolute (per so100_config.py); horizon 16 steps. Values are normalized with per-dataset q01/q99 statistics shipped in experiment_cfg/ and processor/. The policy wrapper applies them; do not feed raw joint values elsewhere.
• Serving: Isaac-GR00T's gr00t/eval/run_gr00t_server.py; SO-101 client: gr00t/eval/real_robot/SO100/eval_so100.py --robot.type=so101_follower. Requires Isaac-GR00T at the pinned commit or later (N1.7).

Data provenance & caveats

• All 39 source repos are public community LeRobot datasets indexed by the Apache-2.0 MolmoAct2 manifest; each source repo carries its own license; the filtered list with repo ids is data/repos_filtered.json.
• Known data issues: several near-duplicate repos (the same dataset re-uploaded by different users), which upweights those scenes; task/scene imbalance across repos was tempered with ds_weights_alpha=0.5.
• Eval limits: offline loss only; expect a sim/real gap. No real-robot rollouts yet.
• Instabilities and failed attempts (lr-6e-4 NaN, mixed-aspect-ratio crash, checkpoint-cadence and GPU-starvation fixes) are documented in the training repo's CLAUDE.md.

Versioning & contact

• main is stable (single published checkpoint, step 21,000). Pin the revision hash for exact reproduction. Superseding checkpoints will set new_version here.
• Issues: HF Discussions on this repo or GitHub issues.