openarm-rby1-gr1-idm-depthanything-v2-large

Multi-embodiment IDM (OpenArm · RB-Y1 · GR-1) — DepthAnything-V2-Large vision tower

One inverse dynamics model (IDM) that predicts joint-space action chunks for three different robots from egocentric video alone. Given frames at t and t+20 (20 Hz, i.e. a 1.0 s window) it predicts the 20-step action chunk that connects them. No state, no language input.

Architecture is GR00T-Dreams' flow-matching IDM with the vision encoder replaced by DepthAnything-V2-Large (DINOv2-Large backbone). The vision tower, projector, VL self-attention and DiT are shared across robots; only the action encoder/decoder are per-embodiment (CategorySpecific weights indexed by embodiment_id). The action space is zero-padded to 66 dims and padded dims are masked out of both the flow-matching input and the loss.

Predecessor (single robot): RyanL22/openarm-idm-depthanything-v2-large (OpenArm only, test MAE 0.0336).

Embodiments

view_ids index the flattened (timestep, view) position [t0L, t0R, t1L, t1R]; the mono robot uses [0, 2] = (t0 left, t1 left) so the same embedding table means the same thing for all three. Do not feed a mono robot with [0, 1].

Action layouts (raw joint order of each robot's dataset; the model output is [20, 66], take the first native dims):

• OpenArm RH56F1 (28): neck 0:2 | left_arm 2:9 | right_arm 9:16 | left_hand 16:22 | right_hand 22:28
• RB-Y1 Wujihand2 (66): wheels 0:4 | torso 4:10 | right_arm 10:17 | right_hand 17:37 | left_arm 37:44 | left_hand 44:64 | head 64:66 (wheels are always 0, torso/head are session constants in the training data; they are kept so the vector equals the raw action column)
• GR-1 Fourier (44): left_arm 0:7 | left_hand 7:13 | left_leg 13:19 | neck 19:22 | right_arm 22:29 | right_hand 29:35 | right_leg 35:41 | waist 41:44 (legs/neck/waist are exactly 0 in the training data)

Per-embodiment q01/q99 normalisation statistics are in experiment_cfg/metadata.json, keyed by tag. Actions are predicted in that normalised space and must be un-normalised with the matching tag's stats.

Inputs

• 224×224 RGB, ImageNet normalisation (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]). SigLIP-range input raises in the tower's input check.
• Training-time augmentation was field-of-view jitter (random zoom-in crop or zoom-out black padding, scale 0.85–1.15, random offset, identical across views/timesteps) plus colour jitter; mirror (left/right) augmentation on OpenArm only. At inference feed the full frame resized to 224×224 (no centre crop).

Training data

Sampling: 1/3 per embodiment per epoch regardless of size (RB-Y1 is up-sampled, GR-1 and OpenArm are sub-sampled).

Training

60,000 steps, global batch 128 (32 × 4), 4×H200, 12 h 54 min, 13.8 mixture epochs. Train loss 1.35 → ~0.005 (mean of last 1,000 steps ≈ 0.007). Vision tower fine-tuned; encoder.layers[11] / vision_model.head frozen as in the predecessor. Initialised from depth-anything/Depth-Anything-V2-Large-hf, not from the single-robot checkpoint.

Results (held-out test splits, step 60,000)

Scored with eval_idm_openarm.py in eval mode (full frame, no augmentation), stride 10, 20-step chunks. "static" = repeat action[t] for 20 steps (the hold-still baseline). MAE in radians; "moving dims" excludes joints that are constant in the data (RB-Y1 wheels/torso/ head, GR-1 legs/neck/waist). "% range" divides the MAE by the training q01–q99 range.

Per group (MAE, radians; IDM / static):

• OpenArm: 0.0350 vs the single-robot predecessor's 0.0336 on the same 34 episodes (+4 %; SigLIP baseline was 0.0359). Hands are on par (0.0360/0.0601 vs 0.0354/0.0608), both arms are slightly worse (0.0249/0.0328 vs 0.0215/0.0303). Joint training plus FoV augmentation cost a little on the robot with the most data.
• GR-1: the only robot where the IDM beats hold-still overall (0.047 vs 0.056); the right (manipulating) arm and hand are clearly better than static, the left side slightly worse.
• RB-Y1: the weakest — arms are ~2× worse than static, and the error is almost entirely in the pose term (0.108 pose vs 0.047 motion for right_arm), i.e. absolute joint angles are misread rather than the motion. 1.47 h of data with 20-dof hands and only 10 test episodes; treat these numbers as preliminary.
• Error is flat across the 20-step horizon for all three (t+1 ≈ t+20), as in the predecessor.

Plots (overlay_ep*.png) and eval_report.json per robot are in the training-side eval directory, not in this repo.

Usage

Code lives in the GR00T-Dreams fork (gr00t/experiment/data_config_idm.py MULTI_CONFIG_MAP["multi3_ae20_depth"], gr00t/data/mixture_dataset.py, IDM_dump/configs/multi3_ae20_depth.yaml). Evaluate one embodiment with its own data config and tag:

python IDM_dump/scripts/eval_idm_openarm.py \
  --checkpoint <this checkpoint> \
  --test-dir  <robot dataset>/test --train-dir <robot dataset>/train \
  --data-config gr1_ego_mono_ae20_depth --embodiment-tag gr1_fourier
# openarm: --data-config openarm_ego_stereo_ae20_depth_mirror_fov --embodiment-tag openarm_rh56f1
# rby1:    --data-config rby1_ego_stereo_ae20_depth              --embodiment-tag rby1_wujihand2

Loading fetches depth-anything/Depth-Anything-V2-Large-hf for the tower definition, so that repo must be reachable or cached.

Files

• model.safetensors, config.json — final weights (step 60,000)
• experiment_cfg/metadata.json — per-tag dataset statistics and modality metadata
• trainer_state.json — loss curve (every 10 steps)
• gate_fov_png/ — example FoV-augmented training frames per robot (pre-flight gate output)