so101-act-sim-pick-and-place

SO-101 ACT Sim Pick-and-Place

Model release: v0.1.0
Training checkpoint: step-200000
Training dataset: wenyixu101/so101-sim-pick-and-place, revision v0.2.0
Status: Experimental, simulation-only

SO-101 ACT Sim Pick-and-Place is an ACT policy for a simulated SO-101 cube pick-and-place task using synchronized front and wrist camera observations. Release v0.1.0 is trained on the new v0.2.0 nominal dataset, which varies cube properties and pose, target-pad location, and external-camera pose.

This checkpoint is published as a reproducible research artifact. It is not a production or real-robot policy.

Released checkpoint

The repository root contains a LeRobot-compatible ACT checkpoint trained from random initialization for 200,000 optimizer steps.

Training data and sampling

The model was trained on wenyixu101/[email protected]:

• 300 new nominal simulation demonstrations
• Training episodes: 0:270
• Validation episodes: 270:300
• Two cube variants: blue 30 mm / 30 g and red 40 mm / 40 g
• Three target-pad position profiles
• Five fixed external-camera pose profiles
• Continuous deterministic cube XY and yaw variation
• Synchronized 640×480 front and wrist camera observations
• No v0.1.x nominal or recovery episodes

Training used a frozen 30-cell sampler over 2 cube × 3 target × 5 camera combinations. Every training cell contributed nine episodes. Each batch contained four blue and four red samples; the complete 200k run drew 1.6 million training samples. The sampler-plan SHA256 was ff92cfe298b381eba6f452b3f833c1094b85592513174f316ecba97c4f2449c6.

The teacher-forced validation mean loss at step 200,000 was 0.0328434351.

Autonomous rollout evaluation

The released 200k checkpoint was evaluated on two independently seeded replicas of the frozen v0.2.0 30-cell holdout. This produced 60 autonomous evaluation episodes. No collection, training, validation, or recovery scene was used.

Evaluation control:

• Maximum 1,200 policy steps per episode; successful episodes may terminate early
• Replanning interval: 10 steps
• Front and wrist camera observations
• so101-viam-50deg-s-v1 action-safety profile

Overall metrics

Terminal outcomes were 13 successes, 36 lifts without target entry, 8 contacts without lift, and 3 target entries without stable release.

Cube-variant metrics

Target and camera metrics

Action-safety observations

• Delta-limited commands: 4,625
• Raw hard-range violations clipped before execution: 1,728
• Maximum raw hard-range excess: 3.395 calibrated units
• Non-finite actions: 0
• Both 30-scene rollout reports passed their frozen acceptance and action-safety gates
• All 120 front/wrist videos across the two 60-episode camera streams decoded successfully

Intended use

This model is intended for:

• Reproducing the Farpoint simulated SO-101 ACT v0.2.0 checkpoint
• Studying generalization across object, target-position, and camera-pose variation
• Comparing imitation-learning checkpoints under frozen autonomous rollouts
• Research and educational experiments in simulation

Limitations

• The model was trained and evaluated only in simulation.
• This release does not demonstrate sim-to-real transfer or real-robot safety.
• The training dataset contains only 300 demonstrations.
• Overall autonomous task success is 21.7%; the model is not reliable enough for deployment.
• Performance is strongly imbalanced: blue success is 11/30, while red success is 2/30.
• Transport remains the dominant failure stage: 36/60 episodes lifted the cube but never entered the target.
• Camera generalization is uneven; the front-X-negative profile produced 0/12 successes.
• Raw policy outputs still require absolute-range clipping and command slew limiting.
• This checkpoint is not suitable for unattended real-robot operation.

Loading with LeRobot

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained(
    "wenyixu101/so101-act-sim-pick-and-place",
)

Use the same observation keys, normalization processors, action ordering, camera setup, and control limits as the training environment.