pi05-BimanualYAM-oranges-uw-notop

π0.5 · Bimanual YAM · wrist-cams-only (no top camera) · "Put all oranges in the bowl"

Full fine-tune of lerobot/pi05_base on brandonyang/yam-ultrawide-teleop — 75 episodes / 108,726 frames of bimanual YAM VR teleoperation with UMI-style ultrawide wrist cameras (312×416 @ 30 fps) — trained without the top camera for an in-the-wild setting. Task string (exact): Put all oranges in the bowl.

Sibling models: pi05-BimanualYAM-oranges (RealSense 3-cam), pi05-BimanualYAM-oranges-uw-top (this dataset with the top camera; pending). Same recipe everywhere: 12,000 steps, global batch 64, seed 1000.

How the top camera is excluded

The camera rename map maps only observation.images.{left,right} → π0.5 slots {left_wrist_0_rgb, right_wrist_0_rgb}. The base-camera slot (base_0_rgb) receives π0.5's native missing-camera treatment: a −1-padded image with a zero attention mask — exactly how openpi pretrained on robots lacking cameras. This is in-distribution for the base model, unlike feeding black frames (which the model would attend to as real content). At inference, provide only the two wrist cameras; no placeholder image is needed.

Interface

• Action: 30-step chunks of 14-D absolute joint positions [left_joint_0..5, left_gripper, right_joint_0..5, right_gripper] at 30 Hz; grippers [0,1].
• Observation: 2 ultrawide wrist RGB cameras + 14-D state (quantile-normalized, discretized into the prompt) + task string.
• Camera mapping and corrected normalization stats ship in this repo's processor pipelines (the hub dataset's stock q01/q99 carry lerobot#4156 aggregation bias of up to ~50% of the quantile range — recomputed globally here).

Training

Identical to the sibling RealSense run (see its card for the full table): fp32 weights + fp32 AdamW (deliberate — LeRobot's bf16 path has no fp32 master weights), cosine 2.5e-5→2.5e-6 with 600-step warmup, quantile normalization, narrowed color jitter + affine augmentation, gradient checkpointing, 8×RTX PRO 6000. 12,000 steps ≈ 7.06 epochs (vs 10.3 for the RealSense run — this dataset has 45% more frames).

Offline eval (final checkpoint, step 12000)

Open-loop 30-step chunk error on training episodes [3, 17, 41, 62, 74], stride 60. All-in-sample; gates brokenness, does not predict task success.

The 8.1× shuffle ratio shows the wrist-only policy is more vision-dependent than the 3-camera variants — with no top view, it must read the wrist cameras. Small boundary overshoots (~1.6k scalars, worst ≈0.02 rad) occur where demonstrations sit exactly on joint/gripper limits; rely on client-side clamping.

Deployment notes

• Cameras named exactly left, right (no top needed). Task string exactly Put all oranges in the bowl. --robot.max_gripper_delta=0.05.
• Local lerobot-rollout: pass --rename_map='{"observation.images.left":"observation.images.left_wrist_0_rgb","observation.images.right":"observation.images.right_wrist_0_rgb"}'.
• Remote policy server: requires the pi05 serving patch (see the sibling model card); the patch also drops the unmapped base slot from the manifest so the client is not asked for a top camera. config.json ships pretrained_path: null intentionally.
• fp32 inference ≈16GB weights → ≥24GB GPU; bf16 autocast at inference is safe.

Training curves: wandb aq-robotics/molmoact2-yam-oranges, run pi05-uw-notop-12k.