pi05-BimanualYAM-oranges-uw-top

π0.5 · Bimanual YAM · ultrawide + top camera · "Put all oranges in the bowl"

Full fine-tune of lerobot/pi05_base on brandonyang/yam-ultrawide-teleop — 75 episodes / 108,726 frames of bimanual YAM VR teleoperation with UMI-style ultrawide wrist cameras (312×416 @ 30 fps) plus the top camera (480×270). Task string (exact): Put all oranges in the bowl.

Sibling models, same recipe (12,000 steps, global batch 64, seed 1000, fp32): pi05-BimanualYAM-oranges (RealSense 3-cam) · pi05-BimanualYAM-oranges-uw-notop (wrist-only, in-the-wild variant).

Interface

• Action: 30-step chunks of 14-D absolute joint positions [left_joint_0..5, left_gripper, right_joint_0..5, right_gripper] at 30 Hz; grippers [0,1].
• Observation: 3 RGB cameras + 14-D state (quantile-normalized, discretized into the π0.5 prompt) + task string.
• Camera mapping ships in the saved processors: observation.images.{top,left,right} → {base_0_rgb, left_wrist_0_rgb, right_wrist_0_rgb} — feed raw dataset-style keys. Corrected normalization stats embedded (the hub dataset's stock q01/q99 carry lerobot#4156 bias up to ~50% of the quantile range; recomputed globally here).

Training

Identical recipe to the siblings (see the RealSense card for the full table): fp32 weights + fp32 AdamW, cosine 2.5e-5→2.5e-6, warmup 600, quantile normalization, narrowed color jitter + affine, gradient checkpointing, 8×RTX PRO 6000. 12,000 steps ≈ 7.06 epochs.

Offline eval (final checkpoint, step 12000)

Open-loop 30-step chunk error on training episodes [3, 17, 41, 62, 74], stride 60. All-in-sample; gates brokenness, does not predict task success.

Offline probes (grasp-event timing, vision/state ablations) show this model and the wrist-only sibling are near-identical — the top camera adds little measurable value on this dataset. Small boundary overshoots (~1.5k scalars, worst ≈0.02 rad) occur where demonstrations sit exactly on joint/gripper limits; rely on client-side clamping.

Deployment notes

• Cameras named exactly top, left, right. Task string exactly Put all oranges in the bowl. --robot.max_gripper_delta=0.05.
• Local lerobot-rollout: pass --rename_map='{"observation.images.top":"observation.images.base_0_rgb","observation.images.left":"observation.images.left_wrist_0_rgb","observation.images.right":"observation.images.right_wrist_0_rgb"}'.
• Remote policy server: requires the pi05 serving patch (see the RealSense sibling card). config.json ships pretrained_path: null intentionally.
• fp32 inference ≈16GB weights → ≥24GB GPU; bf16 autocast at inference is safe.

Training curves: wandb aq-robotics/molmoact2-yam-oranges, run pi05-uw-top-12k.