cimse-so100-bowl-molmoact2-ckpts

⚠️ ERRATUM (2026-09-13). The so100_bowl_* datasets used for this experiment had misaligned language labels: the ManiSkill→LeRobot converter stored episodes in lexicographic key order while per-episode task strings were assigned in numeric order, so ~2/3 of training and validation episodes named the wrong cube colour. Consequences for the results below: (1) the 21 checkpoints were trained on effectively random colour labels and never learned grounding — the 30–60 % "wrong cube" failure rate is this bug, not a policy property; (2) the sim-state GT critical intervals were indexed in raw order and are misaligned with the prediction dumps (the Gemini/VLM intervals were annotated from the dataset videos and are aligned); (3) offline dumps used the same wrong task strings as training. The metric-vs-success correlations are internally consistent for an ungrounded policy, but every conclusion about language, phase attribution and the data-scale family must be treated as void until the zoo is retrained on the relabelled datasets (labels fixed on HF on 2026-09-13). v1 (cimse-so100-experiment, single task, no language) is unaffected.

MolmoAct2 checkpoint zoo — SO-100 "put the {color} cube in the bowl" (ManiSkill3), CI-MSE v2 study

21 MolmoAct2 checkpoints fine-tuned with LeRobot (--policy.type=molmoact2, commit 3f2c29ef) on Kavin60606/so100_bowl_train (2812 sim demos: clean / noisy / strategy-varied oracle). Used to test whether the offline metric CI-MSE predicts closed-loop success. Results, predictions, intervals and the report live in Kavin60606/cimse-so100-experiment-v2.

Runs

Common: batch 16, 3000 steps, seed 1000, chunk 30 @ 30 Hz, absolute joint degrees, normalize_gripper=true, image aug (crop/rotate/colour), bf16, gradient checkpointing, --policy.control_mode="absolute joint pose".

Layout

• <run>/<step>/ — full pretrained_model directory (config, pre/post-processors, model.safetensors, 12.7 GB) for the six anchors: R1_p100/000250 (reference), R1_p100/003000, R2_p05/003000, R3_p15/003000, R4_p40/003000, R5_freeze/003000.
• deltas/<run>_<step>/ — delta for the other 14 fine-tuned checkpoints (zero-shot is the public lerobot/MolmoAct2-SO100_101-LeRobot, not re-hosted; R5_freeze_001500 has no LoRA keys so its "delta" is the full 12 GB): delta.safetensors (only tensors that differ from the reference R1_p100/000250, ~2.9 GB; LoRA is stored unmerged so frozen VLM tensors are bit-identical across checkpoints) plus sidecar/ (config + processor files). Reconstruct with ckpt_delta.py (in cimse-so100-experiment-v2/code_v2):

hf download Kavin60606/cimse-so100-bowl-molmoact2-ckpts --include "R1_p100/000250/*" "deltas/R1_p100_001500/*" --local-dir ck
mv ck/deltas/R1_p100_001500/sidecar ck/deltas/R1_p100_001500/delta.safetensors.sidecar   # unpack expects <delta>.sidecar next to the delta
python ckpt_delta.py unpack ck/R1_p100/000250 ck/deltas/R1_p100_001500/delta.safetensors ck/R1_p100/001500/pretrained_model

Round-trip verified bit-identical (all 1901 tensors) on the box before upload. Uplink limits, not preference, drove the delta format.

Closed-loop success (open-loop chunk execution, 200 episodes, sim)

See cimse-so100-experiment-v2/results/<run>_<step>/ol1|ol2/eval.json; temporal-ensembling control in te/. Headline numbers and the metric correlation are in the report in that repo.