pickplace_skills_vla_v3_2_ckpt060k

SmolVLA — language-conditioned two-disc pick and place (SO-101, MuJoCo)

A SmolVLA policy fine-tuned to pick up a cube and place it on the disc named in a natural-language instruction. Two coloured target discs are visible at all times, so the picture alone cannot say which one is correct — the instruction is the only signal that identifies the goal.

This is the reported final model from the project: the checkpoint at 60,000 steps, about two passes over the dataset. zwan1003/pickplace_skills_vla_v3_2 holds the same run at 90,000 steps, which overfits and scores lower; it is kept for comparison.

Code: the environment, scripted expert controller, data collection pipeline and evaluation harness are at github.com/zwaneiz/so101-vla-pickplace. You need it to reproduce the evaluation, since the two-disc scene is a custom subclass of the so101-nexus environment.

What it does

Use these strings exactly. The policy was trained on them and is sensitive to wording (see Limitations).

Results

Measured on starting positions the model never trained on, averaged over three seeds.

The whole-task score is close to grasp × place. Placement is close to solved; grasping is the bottleneck, caused by a gripper tolerance of under 4 mm on a 25 mm cube.

It follows the words, not the colours. Recolour both discs grey and it still scores 87%. Swap the two colours so they contradict the instruction and it scores 100%, going to the instructed position every time. Remove the direction word and it drops to 3%.

Requirements

python 3.12
lerobot 0.4.4
mujoco 3.9.0
so101-nexus 0.3.12

The environment is a two-disc variant of the so101-nexus pick-and-place scene: a second target disc is injected into the model so both a left and a right target are always present. That subclass, and everything else needed to reproduce the evaluation, is in the code repository.

Loading

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("zwan1003/pickplace_skills_vla_v3_2_ckpt060k")

Four things that will silently break it

1. Camera names must be remapped. SmolVLA expects camera1/camera2; the dataset uses descriptive names. Apply the same mapping used in training:

observation.images.overhead_camera -> observation.images.camera1
observation.images.wrist_camera    -> observation.images.camera2

Get this wrong and the policy receives the wrist image where it expects the overhead view. It does not raise an error, it just behaves badly.

2. Match the physics. The training data was collected with a stiffened contact model and tangential friction scaled by 3.0, representing rubber fingertip pads. Evaluate with the same settings or the grasp will not transfer.

3. Seed torch per episode. SmolVLA samples noise at every inference through its flow-matching action head. Seeding the simulator alone is not enough; without seeding torch, the same command produces different results.

4. Normalisation comes from the dataset. State is normalised on input and actions are de-normalised on output using the statistics of zwan1003/pickplace_skills_v3_2. Omitting this produces smooth but completely misdirected motion.

Limitations

• Narrow language. Near-synonyms are free: grab the cube and pick up the block score the same as the trained wording. Distant phrasing is not: carry the object drops to 7%, colour naming (place it on the blue disc) and negation both fail outright.
• Grasping is the ceiling. The gripper needs better than 4 mm of accuracy. Raising contact friction further lifts whole-task success to about 56% without retraining, which shows the limit is mechanical rather than learned.
• Simulation only. No transfer to a physical arm has been tested or claimed.
• Restricted workspace. Cube positions are drawn from a measured reachable region, not the arm's full range.

Context

Produced for a final-year project investigating whether a compact vision-language-action model, fine-tuned on a small dataset using consumer hardware, genuinely conditions its behaviour on language rather than on visual shortcuts. The whole pipeline — data collection, training and evaluation — ran on a single laptop with a 4 GB GPU.