act_sock_in_bowl_July_9_2026

ACT · Sock-in-bowl · SO-101

An Action Chunking with Transformers (ACT) imitation-learning policy that drives an SO-101 follower arm to pick up a sock and drop it into a bowl. It predicts 100-step action chunks from two camera views plus joint state, and was trained with LeRobot on RyuRobot/Sock-in-bowl-run-2 — 79 teleoperated episodes at 20 FPS.

Camera mapping matters. This policy was trained with two specific views: front = a third-person view of the workspace, wrist = the gripper close-up. At inference the same keys must map to the same physical cameras, or the arm moves smoothly toward the wrong targets. Verify with lerobot-find-cameras opencv before trusting a run.

Model details

• Method: ACT — ResNet-18 vision backbone, transformer VAE, chunk size 100
• Robot: so101_follower (6-DoF)
• Cameras: front (workspace / third-person) and wrist (gripper), both 640×480
• Normalization: MEAN_STD (state, action, visual)
• Framework: LeRobot 0.6.1

Inputs & outputs

Training

Run it on your SO-101 (locally, on a Mac)

ACT is small (~52M parameters), so it runs in real time on Apple-silicon (MPS) — no cloud GPU needed. Plug in the follower (external power on), connect both cameras, then:

PYTORCH_ENABLE_MPS_FALLBACK=1 lerobot-rollout \
  --strategy.type=base \
  --robot.type=so101_follower \
  --robot.port=<your_follower_port> \
  --robot.id=my_awesome_follower_arm \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 20}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 20}}" \
  --policy.path=RyuRobot/act_sock_in_bowl_July_9_2026 \
  --device=mps \
  --fps=20 \
  --task="Sock-in-bowl" \
  --duration=60 \
  --display_data=true

Notes:

• --fps=20 must match the training FPS — the rollout default is 30. Both camera fps values must be 20 too.
• --device=mps overrides the cuda device baked into train_config.json. On a non-Mac GPU box, use --device=cuda.
• Camera indices drift on macOS — confirm which physical camera is which with lerobot-find-cameras opencv, keeping front = workspace and wrist = gripper.
• Port: find it with lerobot-find-port if the arm doesn't connect.
• First run: add --robot.max_relative_target=15 as a safety cap and keep a hand near the power, then remove it (left on, it clamps every action so the arm can't reach its targets).
• --strategy.type=base does not record. To record eval rollouts, use --strategy.type=episodic with a --dataset.* repo.

Evaluation

Not yet evaluated on the real robot. Run it several times per task and record the success rate (successes / trials), noting object positions, lighting, and any distractors.

Citation

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}