cup_pnp

cup_pnp — yellow-cup pick-and-place policies

Four behavior-cloning policies for the task "pick up a yellow cup and place it on a baking tray," trained from egocentric human video reconstructed into robot demonstrations. This is a 2×2 ablation:

• Action / state space: 28-D — right arm only (3-slide + 3-hinge wrist + 22 finger joints), absolute joint targets.
• Observation: single rendered robot-view RGB (224×224).
• sim-tray variant re-renders the tray from its simulation mesh (inpainted over the original footage); the current variant keeps the tray as raw footage. Actions are identical across the two variants — only the observation pixels differ.

Results — held-out action-prediction MSE (normalized, 10-step horizon; lower is better)

π0.5 (pretrained VLA + LoRA) is ~16× more accurate than ABC-DiT (trained from scratch with a randomly-initialized visual backbone). The sim-tray processing is neutral for π0.5 and a small improvement for ABC-DiT.

Contents

• pi05_current_tray/, pi05_sim_tray/ — π0.5 checkpoints (orbax params/ + assets/ norm-stats). Load with openpi policy_config.create_trained_policy. Config: pi05_cup_v1 / pi05_cup_v2.
• abc_dit_current_tray.pt, abc_dit_sim_tray.pt — ABC-DiT checkpoints ({model, step, norm_stats} + optimizer). Load with ABC. state/action dim = 28, camera-keys = top.

Trained on 80 verified reconstructed demonstrations. See the project report for data provenance and method details.