stage3-ur5-actheavy-noaug

Stage 3 — UR5 teleop VLA (Qwen2.5-VL-3B), act-heavy without augmentation

Training run complete: 60,000 steps. Byte-identical setup to Ngseo/stage3-ur5-actheavy except augmentation is off — raw decoded frames go to both the VLM and V-JEPA2, with no ColorJitter/SharpnessJitter and no DomainRandomization.

The config diff against its partner is one line: no_aug: true.

The setup this was trained for

Each of the 7 tasks in Ngseo/ur5_teleop_multitask is only ever shown from one of the 4 cameras, so viewpoint alone almost determines the task. Object colour is deliberately crossed between the two task families so that "colour ⇒ camera" is not a valid shortcut on its own:

Architecture

Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat(8192) → ResNetActionHead → 30×7

L = 1.0·L1(action) + 0.02·InfoNCE(z_a, z_task_future) + 0.002·SIGReg([z_domain_future ; z_b])

Frozen target encoder: Ngseo/stage1 disentangled V-JEPA2 ViT-L, on future frames (8 frames, stride 4 → 1.07 s ahead).

Augmentation ablation — final metrics

Removing augmentation cuts the training action L1 by ~46% and lets InfoNCE fall further. Read this the right way round: augmentation exists to trade training fit for generalisation, so a lower training loss here is the expected consequence of an easier problem, not evidence of a better policy. Whether the augmented model is actually better only shows up out of distribution — asking for a task from a camera it never saw it from — and that measurement is not in this repo.

Reference points on the same normalised scale, none of which use vision or language: dataset mean 0.834, copying the input state across all 30 steps 0.155.

Not evaluated on a robot. Everything above is a training-set loss.

The 7-D joint state is an input here, and a probe on that state alone recovers which of the 7 tasks is running with 94% accuracy (chance 14%), so a viewpoint-OOD failure could not be attributed to the viewpoint shortcut alone.

Related runs

Contents

epoch_6.pt holds model_state_dict (full VLM incl. LoRA), latent_head_state_dict (Head A), free_latent_head_state_dict (Head B), stage2_action_head_state_dict, optimiser state, and the run config. config.yaml is the exact training config.