so101_ACT_graspclothtowasher

Notebook by zexuan

---

📘 README: ACT model; Grasp Experiment (so101_follower)

Background

• Project: Robotic cloth grasping using the LeRobot framework with so101_follower manipulator and 3 cameras.
• Task: Grasp clothes from a naturally cluttered pile and put them into a washing machine.
• Model: ACT policy, trained for 100,000 steps.
• Dataset: 60 episodes, collected in ~3–4 hours.
• Date: 2025-08-19.
• Note: Continued from a previous recording with --resume=true.

---

📝 Data Collection

python -m lerobot.record \
    --robot.disable_torque_on_disconnect=true \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=R12252801 \
    --robot.cameras="{'handeye': {'type':'opencv', 'index_or_path':0, 'width':640, 'height':480, 'fps':30}, 'fixed': {'type':'opencv', 'index_or_path':2, 'width':640, 'height':480, 'fps':30}, 'extra': {'type':'opencv', 'index_or_path':4, 'width':640, 'height':360, 'fps':30}}" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=R07252801 \
    --display_data=false \
    --dataset.repo_id=${HF_USER}/3camera_grasp \
    --dataset.root=/home/zexuan/Robot/data/3camera_grasp00 \
    --dataset.num_episodes=60 \
    --dataset.reset_time_s=2 \
    --dataset.episode_time_s=50 \
    --dataset.push_to_hub=true \
    --resume=true \
    --dataset.single_task="Grasp the cloth to washmachining"

✅ Output:

• Dataset stored at ~/Robot/data/3camera_grasp00.
• 60 episodes successfully collected.
• Dataset pushed to Hugging Face Hub.

---

🏋️‍♂️ Model Training

Train ACT policy for 100k steps:

nohup python scripts/train.py \
  --dataset.repo_id=/home/dreame/zexuan/projects/so101/lerobot/src/lerobot/data/3camera_grasp00 \
  --policy.type=act \
  --output_dir=outputs/train/3camera_grasp \
  --job_name=3camera_grasp \
  --policy.device=cuda \
  --batch_size=16 \
  --steps=100000 \
  --save_freq=5000 \
  --eval_freq=5000 \
  --log_freq=500 \
  > train.log 2>&1 &

📂 Training output directory:

/home/zexuan/Robot/so101/lerobot/src/lerobot/outputs/train/3camera_grasp/checkpoints/100000/pretrained_model

---

☁️ Model Upload

Upload the trained model to Hugging Face Hub:

huggingface-cli upload \
  /home/zexuan/Robot/so101/lerobot/src/lerobot/outputs/train/3camera_grasp/checkpoints/100000/pretrained_model \
  ${HF_USER}/3camera_grasp_pretrained_model_100k

(or use the huggingface_hub Python API to automate uploads).

---

🤖 Model Deployment

Run real robot deployment with multi-camera input:

python scripts/deploy.py \
  --robot.type=so101_follower \
  --robot.disable_torque_on_disconnect=true \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{'handeye': {'type':'opencv', 'index_or_path':0, 'width':640, 'height':480, 'fps':30}, 'fixed': {'type':'opencv', 'index_or_path':2, 'width':640, 'height':480, 'fps':30}, 'extra': {'type':'opencv', 'index_or_path':4, 'width':640, 'height':360, 'fps':30}}" \
  --display_data=false \
  --dataset.single_task="Grasp the cloth to washmachining" \
  --policy.path=/home/zexuan/Robot/so101/lerobot/src/lerobot/outputs/train/3camera_grasp/checkpoints/last/pretrained_model \
  --policy.device=cuda \
  --dataset.episode_time_s=9999 \
  --dataset.repo_id=local/eval_3cameragrasp00 \
  --dataset.push_to_hub=false

📌 Camera mapping:

• 0 → handeye
• 2 → fixed
• 4 → extra

---

✅ Experiment Summary

• Data collection: 60 episodes, successfully uploaded to HF Hub.
• Training: ACT, 100k steps, loss converged well.
• Deployment: Verified on real hardware, stable inference with 3-camera input.
• Conclusion: Multi-view input (3-camera) significantly improves grasping robustness.