smolvla_bun_merged

shawshank1104/smolvla_bun_merged — SmolVLA for the bun pick-and-place, deployment notes for the Thor

Written for the coding assistant on the NVIDIA Jetson AGX Thor. This is the third model for the airoa right arm and the one to deploy for the demo. Everything in the ACT/SmolVLA handoff (README of shawshank1104/act_finalforsure, sections 2–4 and 6) still applies: same input/output contract, same LeRobot 0.6.1 + transformers 5.5.4 + num2words pins, same inference code, same verification protocol, same VLM-cache requirement. Only the differences are listed here.

What changed

Results

Error of the predicted 50-step action chunk vs. the operator, in units of per-joint action std (lower is better). "Held-out" = 40 frames from the 10 unseen episodes; "train" = 20 frames from training episodes.

• On unseen episodes the new model is 44 % better on the first action and 32 % better over the chunk than the previous one. That is the effect of the varied bun data.
• Held-out error plateaued from step 10,000 onward (0.124–0.128 first-step across checkpoints 10k–20k), while training error kept falling. The final checkpoint is as good as any on unseen data and best on seen data, so it is the one shipped. The flow-matching eval_loss in train_log.txt rises over training (0.36 → 0.74); that metric penalizes a sharper model on unseen data and does not reflect the joint-space error above — do not read it as degradation.
• Training loss 1.83 → 0.032, gradient norm 8.0 → 0.40, no NaN, no restarts.

Realistic expectation: first-step error of 12 % of a std on unseen episodes is good; a 33 % chunk error means the open-loop 1.7 s trajectory drifts on novel positions, which closed-loop re-observation every 50 steps corrects. Expect clearly better success than the previous model on bun positions inside the recorded workspace. Positions or lighting outside what the 95 episodes cover remain a gamble.

Download

docker compose -f docker-compose.vla.yml exec vla \
  hf download shawshank1104/smolvla_bun_merged --local-dir /data/checkpoints/smolvla_bun_merged/pretrained_model
# VLM backbone cache, once (same as before; skip if already cached):
docker compose -f docker-compose.vla.yml exec vla hf download HuggingFaceTB/SmolVLM2-500M-Video-Instruct

Verify before moving the arm

docker compose -f docker-compose.vla.yml exec vla python3 \
  /data/checkpoints/smolvla_bun_merged/pretrained_model/verify_policy.py \
  /data/checkpoints/smolvla_bun_merged/pretrained_model cuda shawshank1104/bun_merged

Expected (AWS, must match to the 3rd decimal):

first-step MAE / std (mean over joints): 0.026   50-step: 0.066   finite outputs: True   fingerprint: 28.46926

Inference changes vs. the previous SmolVLA

1. PATH = "/data/checkpoints/smolvla_bun_merged/pretrained_model"
2. task="pick up the bun and put it in the tray" — not "can".
3. Nothing else. Same cameras, same keys, same 30 Hz loop, same policy.reset() per episode, same 50-step chunk, ~0.5–1 s per chunk on the Thor GPU.

Demo-day checklist

• Cameras mounted exactly as during recording; same tray, table height, bun type.
• Run the verify script (above) once after download — it catches every plumbing error offline.
• Do 2–3 rollouts at recorded bun positions first, then the demo positions.
• If the arm hesitates at chunk boundaries, that is the ~0.5 s inference gap; a dummy warm-up inference at startup removes the first-call CUDA delay.
• Fallbacks on the Hub, same contract: shawshank1104/smolvla_finalforsure (fixed position, task "can"), shawshank1104/act_finalforsure (fixed position, ignores task string).

Provenance

Trained 2026-09-15 on AWS g6e.4xlarge (1× L40S), LeRobot 0.6.1, 3 h 49 min. Command in train_config.json; full log in train_log.txt. Datasets: shawshank1104/bun_merged (public) built from shawshank1104/finalforsure (task relabeled can→bun) + shawshank1104/bun (chest camera removed) with LeRobot's lerobot-edit-dataset and merge_datasets.