smolvla-so101-pick-orange

SmolVLA SO101 PickOrange

Fine-tuned SmolVLA policy for the SO101 robot arm performing an orange-picking task in LeIsaac (Isaac Sim).

Task

Pick three oranges from the table and place them on the plate, then reset the arm to rest state. Evaluated in the LeIsaac-SO101-PickOrange-v0 Isaac Sim environment.

Architecture

SmolVLA is a Vision-Language-Action model that combines a frozen vision encoder with a language model backbone and a lightweight action expert head.

Inference pipeline

2 camera images (480x640)
  -> resize to 512x512 with padding
  -> patchify into 16x16 patches (1024 tokens per image, 2048 total)
  -> 12-layer ViT vision encoder (bf16)
  -> project to LM hidden space
  -> 16-layer SmolLM2 backbone + 16-layer Expert (interleaved cross-attention)
  -> decode 50 action tokens -> 6D joint positions

Vision Encoder (ViT)

Text/LM Backbone (SmolLM2)

Action Expert Head

Full Model Summary

Branches

Training

Usage

Serve the policy

python -m lerobot.scripts.serve \
  --policy.type=smolvla \
  --policy.pretrained_path=edge-inference/smolvla-so101-pick-orange \
  --policy.vlm_model_name=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
  --port=8080

Evaluate in Isaac Sim (requires LeIsaac + Isaac Sim)

python scripts/evaluation/policy_inference.py \
  --task=LeIsaac-SO101-PickOrange-v0 \
  --policy_type=lerobot-smolvla \
  --policy_host=localhost \
  --policy_port=8080 \
  --policy_checkpoint_path=edge-inference/smolvla-so101-pick-orange \
  --policy_action_horizon=50 \
  --policy_language_instruction="Pick up the orange and place it on the plate" \
  --eval_rounds=10 \
  --device=cuda \
  --enable_cameras

Use the single-rank branch (lower loss)

from huggingface_hub import snapshot_download
snapshot_download(
    "edge-inference/smolvla-so101-pick-orange",
    revision="single-rank",
    local_dir="./checkpoint-single-rank"
)

Dataset

The training data was collected via teleoperation inside the LeIsaac simulation (Isaac Sim), meaning there is zero visual domain gap between training and evaluation environments.

Files

• model.safetensors -- Model weights (1.2 GB)
• config.json -- Policy architecture config
• train_config.json -- Full training configuration (reproducible)
• policy_preprocessor*.json/safetensors -- Input normalization (state mean/std)
• policy_postprocessor*.json/safetensors -- Output denormalization (action mean/std)