act-so100-grab-to-bowl

ACT SO-100 Grab-to-Bowl

This project trains and evaluates an ACT policy for a real-world robotic manipulation task using Hugging Face LeRobot and a SO-100/SO-101 robotic arm setup.

Task

The robot learns to complete the task:

Grab the object and put it into the bowl.

The full workflow includes hardware setup, leader-follower teleoperation, camera-based data collection, policy training, and real-robot autonomous inference.

Hardware

• Robot: SO-100/SO-101 leader-follower robotic arm setup
• Motors: Feetech STS3215 serial bus servos
• Follower arm port: COM9
• Leader arm port: COM10
• Camera: OpenCV camera index 0
• Image resolution: 640x480
• FPS: 30

Dataset

The dataset was collected through real-world teleoperation demonstrations.

• Dataset name: local/grab_to_bowl
• Number of demonstration episodes: 76
• Task: Grab the object and put it into the bowl
• Observation: front camera image + robot joint state
• Action: 6-DoF joint position action

The dataset includes an additional set of targeted demonstrations focused on improving the release phase:

move above bowl -> open gripper fully -> pause -> lift empty gripper

These extra demonstrations were added after observing that earlier policies could sometimes grasp the object but fail to release it into the bowl.

Model

The main successful policy is ACT.

• Policy type: act
• Training steps: 20,000
• Checkpoint: act_grab_to_bowl_full76/checkpoints/020000/pretrained_model
• Model weights: model.safetensors

Real-Robot Evaluation

The trained ACT policy was evaluated on the physical robot.

• Total evaluation runs: 10
• Successful runs: 8
• Success rate: 80%

Success means the robot autonomously completes:

locate object -> grasp object -> move to bowl -> release object into bowl

The main failure mode was:

The first grasp missed the object and pushed it away from the trained start region. The policy then kept attempting to grasp, but the object had moved outside the training distribution and the run timed out.

Demo

The real-robot evaluation video is included at:

demo/act_eval_10eps_merged.mp4

ACT vs Diffusion Policy

Diffusion Policy was also trained on the same 76-episode dataset for comparison.

• Diffusion training steps: 20,000
• Final training loss: approximately 0.017 to 0.021
• Real-robot behavior: unstable; the gripper mostly showed small tremors and did not execute a complete grasping trajectory

This comparison suggests that, for this small real-world dataset and fixed manipulation task, ACT was more data-efficient and reliable than Diffusion Policy. Diffusion may require more demonstrations, broader task coverage, and more consistent state-action trajectories to work well on the real robot.

Files

Recommended model repository structure:

act-so100-grab-to-bowl/
├─ README.md
├─ config.json
├─ model.safetensors
├─ policy_postprocessor.json
├─ policy_postprocessor_step_0_unnormalizer_processor.safetensors
├─ policy_preprocessor.json
├─ policy_preprocessor_step_3_normalizer_processor.safetensors
├─ train_config.json
├─ demo/
│  └─ act_eval_10eps_merged.mp4
└─ scripts/
   ├─ eval_model.py
   ├─ record_data.py
   └─ train_act.py

Local Scripts

Evaluate ACT

python eval_model.py --num_episodes 1 --model outputs/train/act_grab_to_bowl_full76/checkpoints/020000/pretrained_model

Record More Demonstrations

python record_data.py --num_episodes 10 --episode_time 25 --resume

Notes

The tutorial documents used during learning are not included in this repository. This repository only contains the author's own model, scripts, evaluation video, and project summary.