smolvla-pick-place-violet-block

SmolVLA Robot Policy

Trained on an SO-101 robot. Pick and place of a violet cube into a white square.

Camera 0: workspace camera

Camera 1: hand camera

Camera 2 is not used

<figure> <img src="https://cdn-uploads.huggingface.co/production/uploads/69b83617979e455a4b0bd61b/AbYTcEFIRwvpdfCY5uRGo.jpeg" width="500"> <figcaption><em>Model training setup. The workspace camera is on the top of the black arm and has the mat filling the field of view.</em></figcaption> </figure>

Some sample prompts for better interfacing succes:

• Use the workspace camera to locate the violet block. Ignore shadows, reflections, printed text, and markings on the mat. Estimate the block's position using only its three-dimensional shape and violet color. Move the arm toward the block while keeping it visible in the workspace camera whenever possible. If the arm blocks the workspace camera or the block leaves its view, switch to the hand camera and continue tracking the same block. Center the block in the hand camera, align the gripper, grasp it, and use the hand camera to confirm that the block is securely held. Move to the white square, place the block inside the square, release it, and verify with the hand camera that the block is no longer in the gripper.
• Find the violet block using the workspace camera. Do not mistake shadows, lighting changes, or printed text for the block. If the workspace camera becomes occluded by the robot arm, rely on the hand camera to continue tracking the block instead of searching for a new object. After grasping, verify in the hand camera that the block is between the gripper fingers before moving to the white square. Place the block inside the white square and release it.
• Locate the violet cube in the workspace camera. Treat shadows, reflections, and printed graphics on the mat as background and ignore them. Track the physical cube rather than image features on the table surface. As the arm approaches, use the workspace camera while the cube is visible. If the cube becomes occluded by the robot arm, continue tracking it using the hand camera. Use the hand camera for the final alignment, grasp confirmation, and placement into the white square.