molmoact2_lora_v1_1

MolmoAct2-LIBERO LoRA — no-augmentation run (8,000 and 16,000 steps)

日本語版: README.ja.md

A LoRA adapter trained on top of allenai/MolmoAct2-LIBERO-LeRobot for the LIBERO manipulation benchmark, using a demonstration set rebuilt to reduce two specific failure modes: contact with non-target objects, and a narrow distribution of robot initial poses.

Trained for the PARC2026 robot-manipulation competition (finals round 1).

Read this before using it. This run was trained with image augmentation switched off (image_transforms.enable: false) — the base model's own training recipe uses it, and turning it off here was a mistake, not a design choice. The run was stopped at 16,000 steps and restarted from scratch with augmentation on; that corrected run is published separately as siruku6/molmoact2_lora_v1 (24,000 steps) and siruku6/molmoact2_lora_v1_40k (40,000 steps). If you want the better model, use those. This repository is published so the weaker arm of the comparison is inspectable rather than only described.

What this is

Weights are saved unmerged, with PEFT parameter names (base_model.model.*, .base_layer.*). Loading requires peft and enable_lora_vlm: true in the policy config.

Training

steps          40000 (planned; stopped at 16000)
batch_size     2
optimizer      adamw, lr 1e-5, weight_decay 0.0, betas (0.9, 0.95), eps 1e-6
               grad_clip_norm 1.0
scheduler      cosine_decay_with_warmup
               warmup 200, decay 40000, peak_lr 1e-5, decay_lr 1e-6
seed           1000
chunk_size     10      n_action_steps 10

Image augmentation was off (image_transforms.enable: false). The corrected run uses up to 3 of: brightness 0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, affine ±5° with ±5% translation — matching the augmentation the base model itself was trained with.

Checkpoints were saved every 8,000 steps. This repository holds 008000/ and 016000/.

The full training log is in logs/train.log; logs/train_metrics.log is the same log reduced to the per-log_freq metric lines (step, loss, grad norm, learning rate, timings).

Evaluation

Only the 16,000-step checkpoint was scored. Measured on 84 perturbed LIBERO task rows × 5 episodes = 420 episodes (EGL rendering, samples=1, no adaptive replanning). "Collision rate" excludes scene artifacts that no policy can avoid.

Read this carefully:

• This is not evidence that augmentation helps. Three things differ between the two runs: augmentation off → on, batch 2 → 4, and 16,000 → 24,000 steps. It is not a single-variable comparison, so the 0.695 → 0.774 gap cannot be attributed to any one of them.
• The evaluation set contains no camera-viewpoint, background, or lighting perturbations. It is 12 base rows, 60 sensor-noise rows, and 12 robot-initial-pose rows. Conclusions about viewpoint or appearance robustness cannot be drawn from it — which is exactly where missing augmentation would be expected to show.
• The 8,000-step checkpoint was never scored.
• The unmodified base model outscored this lineage on the competition's own scoring runs (0.530 and 0.424 on two tracks).

Usage

from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
from lerobot.configs.policies import PreTrainedConfig

config = PreTrainedConfig.from_pretrained("path/to/checkpoint")
config.checkpoint_path = "path/to/checkpoint/molmoact2_base_small"
policy = MolmoAct2Policy.from_pretrained("path/to/checkpoint", config=config, strict=True)

peft must be installed. molmoact2_base_small/ (config, tokenizer, normalization stats) is included in this repository and is required.

Running without network access

The policy config names a Hub repository for the discrete action tokenizer (discrete_action_tokenizer: "allenai/MolmoAct2-FAST-Tokenizer"). At load time LeRobot resolves that string as a local path first, and falls back to snapshot_download if no such path exists. In an offline environment the fallback fails and the policy will not start.

A copy of that tokenizer is bundled here as fast_action_tokenizer/, byte-identical to the upstream one. Point the config at it and no network is needed:

config.discrete_action_tokenizer = "path/to/checkpoint/fast_action_tokenizer"

We lost a competition submission to exactly this (the server could not start; score 0), which is why it ships with the weights. Any config that still names a Hub repository will fail the same way — check every string in policy_preprocessor.json before relying on an offline run.

License and provenance

Training data is not included here; only the resulting weights. See NOTICE for the full provenance chain and LICENSE for the Apache 2.0 text.

Related

• Corrected run, 24,000 steps: siruku6/molmoact2_lora_v1
• Corrected run, 40,000 steps: siruku6/molmoact2_lora_v1_40k
• Dataset: siruku6/libero_combined_bowl5_initpose_l90clean
• Base model: allenai/MolmoAct2-LIBERO-LeRobot