so101_pick_place_pen_smolvla

so101_pick_place_pen_smolvla

A SmolVLA vision-language-action policy fine-tuned with LeRobot for a pick-and-place pen task on the SO-101 robot arm.

이 모델은 SO-101 로봇 팔에서 펜 집어 옮기기(pick-and-place) 작업을 수행하도록 LeRobot으로 파인튜닝한 SmolVLA 비전-언어-액션 정책입니다.

Model Details / 모델 정보

• Policy / 정책: SmolVLA (fine-tuned from lerobot/smolvla_base)
• Robot / 로봇: SO-101 follower arm
• Task / 작업: Pick and place a pen / 펜 집어 옮기기
• Dataset / 데이터셋: nota-gmbh/pick_and_place_pen_so101
• Episodes / 에피소드: 53

Training / 학습

• Steps: 40,000
• Hardware: Multi-GPU
• Batch size: 64
• Base model: lerobot/smolvla_base

During data collection, the camera frames were captured keeping a consistent width-to-height aspect ratio across episodes for visual consistency.

데이터 수집 시, 에피소드 전반에 걸쳐 카메라 프레임의 가로:세로 비율을 동일하게 유지하려고 노력했습니다.

export NCCL_P2P_DISABLE="1"
export NCCL_IB_DISABLE="1"
export HF_USER="~~"
export TASK_NAME="pick_and_place"

accelerate launch --multi_gpu --num_processes=4 \
  $(which lerobot-train) \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=Dangni/pick_and_place_so101_pen \
  --dataset.video_backend=pyav \
  --dataset.image_transforms.enable=true \
  --dataset.image_transforms.max_num_transforms=3 \
  --dataset.image_transforms.random_order=true \
  --batch_size=16 \
  --num_workers=1 \
  --steps=100000 \
  --output_dir=outputs/train/so101/smolvla/pick_and_place_$(date +%Y%m%d_%H%M%S) \
  --job_name=smolvla_yellow \
  --policy.device=cuda \
  --policy.freeze_vision_encoder=true \
  --policy.train_expert_only=true \
  --policy.push_to_hub=false\
  --policy.repo_id=Dangni/pick_and_place_yellow_smolvla \
  --rename_map='{"observation.images.fixed": "observation.images.camera1", "observation.images.handy": "observation.images.camera2"}' \
  --save_freq=5000 \
  --wandb.enable=true

Optimizer / 옵티마이저

Note: The learning rate decays over training via the scheduler, so 3.2594e-05 reflects a logged intermediate step rather than the final value. 참고: LR은 스케줄러에 의해 학습 중 감소하며, 3.2594e-05는 로깅된 중간 스텝의 값입니다.

Usage / 사용법

lerobot-rollout `
  --strategy.type=base `
  --policy.path=Dangni/pick_and_place_smolvla_so101_040000 `
  --policy.n_action_steps=50 `
  --robot.type=so101_follower `
  --robot.port=COM3 `
  --robot.id=follower_arm `
  --robot.cameras='{fixed: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, handy: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}' `
  --rename_map='{"observation.images.fixed": "observation.images.camera1", "observation.images.handy": "observation.images.camera2"}' `
  --task="Pick the pen and place it inside the yellow border" `
  --duration=500

Limitations / 한계

• Trained on a single pick-and-place pen task with 53 episodes; performance is limited to this task and the data-collection setup.
• Camera resolution and viewpoint should match the training setup for reliable inference.
• 단일 펜 집기 작업, 53개 에피소드로 학습되어 해당 작업과 수집 환경에 한정됩니다.
• 추론 시 카메라 해상도·시점을 학습 환경과 동일하게 맞춰야 안정적으로 동작합니다.

Links / 링크

• Base model: lerobot/smolvla_base
• Dataset: nota-gmbh/pick_and_place_pen_so101
• LeRobot: https://github.com/huggingface/lerobot aa