smolvla-so101-pickplace

smolvla-so101-pickplace — language-conditioned pick-and-place

Successor: smolvla-so101-pickplace-v2 reaches 90% on the same protocol (trained on the 160-episode -v2 recovery set, which fixed this model's right-side coverage failures).

SmolVLA (450M) fine-tuned on ahmedsohail2003/so101-sim-pickplace — 100 language-labeled MuJoCo demonstrations of an SO-ARM100 arm performing:

"Pick up the red block and place it in the blue tray."

Results — measured, same protocol for every row

20 evaluation episodes in the MuJoCo work-cell (fixed eval seed disjoint from training data, nominal scene, 240-step cap at 15 Hz):

Honest read: 12k steps of expert-only fine-tuning takes a generalist VLA from 0% to within 10 points of a specialist behavior-cloning baseline on the same data — while being commandable in natural language. Failures are non-engagements clustered on right-side block spawns (under-represented in the 100 demos), not failed grasps: the policy either commits and succeeds consistently, or never engages.

Fine-tuning recipe (free-tier: single Kaggle T4)

• LeRobot 0.6.0, lerobot-train, AMP fp16, batch 8, 12k steps (~8 h)
• Frozen VLM backbone, action expert only (freeze_vision_encoder=True, train_expert_only=True) → ~100M trainable of 450M total
• lr 1e-4 peak, cosine decay, warmup 1k; loss 0.97 → ~0.06
• Camera mapping from the pretrained base's naming: --rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.wrist": "observation.images.camera2"}' (2 dataset cameras are a subset of the base's 3; the absent third is skipped)

Use

import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

repo = "ahmedsohail2003/smolvla-so101-pickplace"
policy = SmolVLAPolicy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(
    policy_cfg=policy.config, pretrained_path=repo,
    preprocessor_overrides={"device_processor": {"device": "cuda"}},
)

batch = {
    "observation.state": state_6d,                      # (1, 6) float32
    "observation.images.camera1": front_rgb,            # (1, 3, 224, 224) in [0, 1]
    "observation.images.camera2": wrist_rgb,            # (1, 3, 224, 224) in [0, 1]
    "task": "Pick up the red block and place it in the blue tray.",
}
action = post(policy.select_action(pre(batch)))         # (1, 6) joint setpoints

Evaluation script: eval_smolvla.py in the TalkToTheCell project (env + protocol from the sim2cell work-cell).

Provenance

• Base: lerobot/smolvla_base (Apache-2.0)
• Data: author-recorded scripted-expert demonstrations in MuJoCo (dataset card)
• Robot model: MuJoCo Menagerie trs_so_arm100 (Apache-2.0)