basevla-maniskill-aug

BaseVLA — ManiSkill Franka (Qwen2.5-VL-3B), vision-only, with augmentation

Plain behaviour cloning baseline: no V-JEPA, no latent alignment, no decorrelation. Just L1(action) on ManiSkill's merged Franka set, trained without proprioception and with domain augmentation on.

Checkpoint at step 40,000. The run was stopped early at ~44k steps (of a planned 100k) because the loss had flattened — 0.0487 at 30k to 0.0474 at 42k.

Data

maniskill-franka-merged from Ngseo/stage1-data, single camera (observation.images.anchor, 256×256 h264), 30 fps, action action.ee_delta_pose (7-D).

The raw set needs filtering. 17.1% of its action.ee_delta_pose rows are NaN, and they are not spread evenly — whole tasks were never populated:

The ~1% floor is the last frame of each episode (no next action) and is normal. PushT has no usable action under any of the three control modes.

Training on it unfiltered silently produces a broken model: NaNs get zeroed by nan_to_num, so the policy learns "output nothing" on 17% of frames, and the normalisation statistics come out as NaN.

This run therefore drops episodes whose action column is NaN in more than 5% of rows (--max_nan_frac 0.05):

9,216 / 12,257 episodes kept  →  1,218,188 frames (83.3% of the original)

Architecture

Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat → ResNetActionHead → 30×7

L = L1(action)        ← that is the whole objective

--no_state is used, so the state token degenerates to a constant learned bias (state_proj(0)) and no proprioception reaches the model. ManiSkill's merged set has no observation.state column to begin with.

Training curve

Not evaluated in simulation or on a robot. The number above is a training-set L1 on normalised actions. No ManiSkill success rate is reported.

Contents

checkpoint.pt holds policy (full VLM incl. LoRA), latent_head, free_latent_head, action_head, optimizer, agg_stats (normalisation statistics) and args (the exact CLI configuration, including the NaN filter threshold).

import torch
ck = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)
ck["step"]        # 40000
ck["args"]        # full training config
ck["agg_stats"]   # action normalisation used at train time