act_banana_in_pot

ACT — Put the right banana in the pot (UR7e)

An Action Chunking Transformer (ACT) policy trained by imitation learning to perform the manipulation task "put the right banana in the pot" on a Universal Robots UR7e arm with two RGB cameras.

• Policy: ACT (Transformer encoder–decoder with a CVAE), ResNet18 vision backbone (ImageNet-pretrained), chunk_size = 100, n_action_steps = 100.
• Trained on: Bigenlight/banana_in_pot_lerobot_v3 (51 teleoperated episodes / 21,524 frames, UR7e follower + GELLO leader, 2 cameras).
• Training length: 50,000 steps, batch size 8, AdamW lr = 1e-5 (constant).
• Framework: LeRobot v0.6.1.

What the policy does

Images were captured at 720p and resized on-the-fly to 360×640 during training (aspect-preserving half-resolution; no dataset re-encode). You must resize every live camera frame to 360×640 at inference — a different aspect ratio or interpolation degrades the policy.

Data & hardware setup

Training configuration

Training loss

Final training loss ≈ 0.065.

Offline evaluation (open-loop: predicted actions vs. ground truth)

Evaluated on held-out episodes (train vs. held-out gap is negligible → generalizes, no overfitting). The step 50,000 checkpoint (this model) is the best by held-out L1.

Best checkpoint (step 50,000):

• Held-out overall L1 = 0.0235 rad (≈ 1.34°); train L1 = 0.0205 → near-zero gap.
• Joints MAE = 0.0237 rad (≈ 1.36°).
• Gripper accuracy 99.2%.

The wrist joint (cmd6) carries the largest error — it is the axis with the most natural variation. The gripper channel is close to binary.

How to use

Load the policy (LeRobot)

from lerobot.policies.act import ACTPolicy
from lerobot.policies import make_pre_post_processors

policy = ACTPolicy.from_pretrained("Bigenlight/act_banana_in_pot")
policy.eval()

# Normalization is NOT baked into forward() in lerobot 0.6.1 — it lives in the
# processor pipeline saved alongside the checkpoint. select_action returns a
# NORMALIZED action; the post-processor converts it back to radians.
preprocessor, postprocessor = make_pre_post_processors(
    policy_cfg=policy.config,
    pretrained_path="Bigenlight/act_banana_in_pot",
)

Per control tick:

policy.reset()                       # once at the start of each rollout
obs = preprocessor(obs)              # normalize + batch + move to device
action = policy.select_action(obs)   # normalized
action = postprocessor(action)       # radians, numpy (7,)

select_action returns one action per call from an internal queue; on an empty queue it predicts a full 100-step chunk and replans after 100 executed actions (temporal ensembling is off by default in this config).

Deploy on a real UR7e

LeRobot ships no UR robot class — you build the observation dict yourself and stream joint targets with ur_rtde at 30 Hz (33.3 ms control period). Loop: read getActualQ() + gripper → grab both cameras, BGR→RGB, resize to 360×640 → build observation.state / observation.images.cam1 / cam2 → preprocess → select_action → postprocess → servoJ(q_target[:6]) + drive gripper from grip_cmd.

⚠️ Safety — actions are ABSOLUTE joint positions (the single biggest safety driver):

1. Start near the dataset initial pose q1..q6 ≈ [2.84, -1.41, 1.78, -2.01, -1.66, -3.42] rad before enabling the policy, or the first absolute command is a large jump.
2. First-command jump guard: if max(|q_target − getActualQ()|) > ~0.15 rad, abort.
3. Clamp per-tick joint change (e.g. ≤ 0.05–0.1 rad / 33 ms early on) and clamp to UR software joint limits.
4. Reduced speed for first trials; keep a hand on the E-stop dead-man switch.
5. Camera mapping matters: the policy learned a fixed cam1/cam2 → physical-viewpoint mapping. Swap them and it fails silently. Verify wiring every session.
6. Gripper grip_cmd is ~binary — threshold (e.g. >0.5 → close) and map to your driver.

A full deployment guide (with a reference ur_rtde skeleton and all the citations to the lerobot inference/normalization code paths) is in the project's DEPLOY_UR.md.

Related repositories

• Dataset: Bigenlight/banana_in_pot_lerobot_v3
• HIL-SERL prep bundle (reward classifier, SAC config, runbook) for taking this task online with reinforcement learning: Bigenlight/banana_in_pot_hilserl

Limitations

• Small dataset (51 demos). Metrics reflect an easy, temporally-correlated task; expect reduced robustness to novel object positions, lighting, or camera placement.
• Offline metrics only. All numbers above are open-loop (predicted vs. ground-truth actions). Real closed-loop task success on the arm has not been measured here — a physical UR7e deployment is required for the final verdict.
• Absolute-joint action space demands the safety guards above; the policy was only ever conditioned on states near the data-collection start pose.
• The ResNet18 backbone is ImageNet-pretrained (not robotics-pretrained); the CVAE and transformer are trained from scratch on this task.