octo_so101_tape

Octo-Small 1.5 finetuned on SO-101 "put the tape into the box"

Octo-Small 1.5 (27M) converted from JAX to LeRobot and finetuned on a 10-episode SO-101 tape-pickup dataset.

Language-conditioned, two cameras (front 256x256, wrist 128x128), 2-step observation history, 4-action chunk from a diffusion head (20 DDPM steps). No proprioception — Octo-1.5 has no state tokenizer, so observation.state is not consumed.

Requirements

Needs a LeRobot build that has the octo policy registered (policy.type == "octo"). The frozen google-t5/t5-base encoder is not stored here — it is downloaded from the Hub when the policy is instantiated, which is why this repo is 104 MB rather than ~550 MB.

Finetuning command

# 1. Convert the released JAX checkpoint to LeRobot (once)
lerobot-convert-octo \
  --checkpoint hf://rail-berkeley/octo-small-1.5 \
  --output outputs/octo_small_1.5 \
  --image-key primary=observation.images.front \
  --image-key wrist=observation.images.wrist

# 2. Finetune
lerobot-train \
  --dataset.repo_id=khanhnd61/so101-tape_20260804_224429 \
  --policy.path=outputs/octo_small_1.5 \
  --policy.device=cuda \
  --policy.push_to_hub=false \
  --steps=20000 \
  --batch_size=16 \
  --num_workers=8 \
  --log_freq=100 \
  --save_freq=5000 \
  --seed=1000 \
  --wandb.enable=false \
  --output_dir=outputs/train/octo_so101_tape

Batch 16 is the ceiling on a 12 GB card (6.9 GB used; batch 32 OOMs). SO-101 is 6-dim while the pretrained head is 7-dim, so reverse_network.linear1.weight, linear2.weight and linear2.bias are re-initialized on load — expected, everything else loads strictly.

Training

20,000 steps (93 epochs over 3,434 frames), 2h58m on an RTX 3060, ~30 samples/s.

Open-loop accuracy

60 frames spread over all 10 episodes, fresh reset each (no history), against recorded actions:

per-joint MAE : [0.558 1.937 1.481 0.755 0.456 0.293]
per-joint |gt|: [51.9  53.1  51.2  60.3  147.3  4.2 ]
overall MAE   : 0.9133   (1.49% of mean |action|)

Caveat: 93 epochs on 10 episodes with no held-out split. These numbers are consistent with a policy that has memorized the demonstrations; they say nothing about generalization to a new tape position.

Rollout

lerobot-rollout \
  --strategy.type=base \
  --policy.path=khanhnd61/octo_so101_tape \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_awesome_follower_arm \
  --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
  --task="Put the tape into the box" \
  --duration=60

No --rename_map: front/wrist are baked into the checkpoint's image_keys. Keep --task byte-identical to the recorded task — Octo conditions on it.

Latency: 24 ms per 4-action chunk on an RTX 3060, 170 ms with torch on an i5-12400F. The chunk is only 4 actions, so at 30 fps a result is needed every 133 ms and n_action_steps cannot exceed chunk_size=4 without retraining. CPU inference does not keep up.