smolvla_bunv2

shawshank1104/smolvla_bunv2 — SmolVLA fine-tuned on the demo-site data. Deploy THIS one.

Written for the coding assistant on the NVIDIA Jetson AGX Thor. Everything from the smolvla_bun_merged README still applies (contract, pins, inference code, VLM cache, verification). This file lists only what is different, plus two inference-side changes that are required for reliable grasping.

Why this model exists

At the demo site the previous model (smolvla_bun_merged) aligned the arm correctly but did not grip. Offline analysis on held-out grasp events reproduced it: the policy closed the gripper (joint 8) on only 2 of 5 approaches, often stopping half-way (0.45, 0.84) or not at all (0.05). Two causes: (1) the close decision depends on a fine visual cue the earlier data covered thinly; (2) with 50 executed steps per chunk the closing time is committed 1.7 s ahead, so the policy cannot react to contact.

What changed in training

Results (gripper closure on the 3 held-out on-site episodes, and verification numbers) are in the Results section at the bottom, appended after training.

REQUIRED inference changes on the Thor

1. Re-observe every 25 steps instead of 50 (config override, no retraining)

from lerobot.configs.policies import PreTrainedConfig
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from lerobot.policies.factory import make_pre_post_processors

PATH = "/data/checkpoints/smolvla_bunv2/pretrained_model"
cfg = PreTrainedConfig.from_pretrained(PATH)
cfg.n_action_steps = 25            # chunk_size stays 50; execute the first 25, then re-infer
cfg.device = "cuda"
policy = SmolVLAPolicy.from_pretrained(PATH, config=cfg).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(policy.config, pretrained_path=PATH,
                                 preprocessor_overrides={"device_processor": {"device": "cuda"}})

Effect: the policy sees the bun under the fingers 0.8 s later at the latest and can commit the close. Cost: one inference per 25 frames (~0.5 s on the Thor per call, once every 0.83 s of motion) — still fits, but do a warm-up inference at startup and keep the loop at 30 Hz. If the Thor cannot sustain it (visible stutter at every chunk boundary), fall back to 50 and rely on change 2 alone.

2. Snap the gripper command (post-processing, no retraining)

In the demos joint 8 is binary: 0 = open, 1 = closed, almost never in between. A half-closed command is never what the operator did. After postprocessor(...) and before sending to the arm:

GRIPPER = 7                                  # index of right_j8.pos in the 8-dim action
a = action.clone()
a[..., GRIPPER] = torch.where(a[..., GRIPPER] > 0.5, torch.ones_like(a[..., GRIPPER]), torch.zeros_like(a[..., GRIPPER]))

Use 0.5 as the threshold. If the gripper still releases early during transport, lower it to 0.4; if it closes on nothing, raise to 0.6. Keep the other seven joints untouched.

3. Everything else unchanged

Same cameras and keys, RGB, 30 Hz, policy.reset() per episode, task string exactly one of the two above. For a multi-bun demo use "pick up all the buns and put them in the tray" — 25 of the 33 on-site episodes were recorded under that task.

Download and verify

docker compose -f docker-compose.vla.yml exec vla hf download shawshank1104/smolvla_bunv2 --local-dir /data/checkpoints/smolvla_bunv2/pretrained_model
docker compose -f docker-compose.vla.yml exec vla python3 /data/checkpoints/smolvla_bunv2/pretrained_model/verify_policy.py /data/checkpoints/smolvla_bunv2/pretrained_model cuda shawshank1104/bun_merged_v2

Expected numbers: see Results below (must match to the 3rd decimal).

Demo-day order of operations

1. Verify script above → numbers match.
2. Dry run with the arm disabled on one on-site episode through the live pipeline (checks cameras, keys, RGB, joint order).
3. Two rollouts at recorded on-site bun positions, watching the gripper: it should close fully within ~1.5 s of arriving. If it hesitates: confirm change 1 and 2 are active, then lower the snap threshold to 0.4.
4. Demo positions.
5. Fallback on the Hub with the same contract: shawshank1104/smolvla_bun_merged (aligns, weak grasp) — only with change 2 applied.

Results (auto-appended after training)

Gripper closure on 3 held-out on-site episodes, per checkpoint (baseline smolvla_bun_merged: 2/5 closes):

checkpoints: closing events 5 | policy closes gripper: 4/5 | close timing vs operator (frames, +late): median -4 | arm joints MAE during approach (rad): 0.054
checkpoints: closing events 5 | policy closes gripper: 4/5 | close timing vs operator (frames, +late): median 0 | arm joints MAE during approach (rad): 0.057
checkpoints: closing events 5 | policy closes gripper: 4/5 | close timing vs operator (frames, +late): median 0 | arm joints MAE during approach (rad): 0.059
checkpoints: closing events 5 | policy closes gripper: 4/5 | close timing vs operator (frames, +late): median -1 | arm joints MAE during approach (rad): 0.059

Verification on bun_merged_v2 training frames (Thor must reproduce):

  first-step MAE / std (mean over joints): 0.043   50-step: 0.095
  finite outputs: True   latency per chunk on cuda: median 235.4 ms
  fingerprint (sum of first-step preds): 30.1566