act_so101_cubcyl_poshold_chunk50_bc_3cam

ACT without CVAE (behavioural cloning) - SO-101 - 3 cameras - chunk 50

Action Chunking Transformer with the CVAE objective disabled (use_vae=false) — no latent, no KL term, plain L1 regression onto action chunks — trained on all three camera feeds. 34.20M parameters against 51.57M for the CVAE variant.

Baseline for act_so101_cubcyl_poshold_chunk50_cvae_3cam, identical in every other respect.

Final eval_loss on the held-out 30 episodes: 0.2067 — the highest of six variants, though the whole spread is within noise.

One observation worth recording

This is the only variant whose eval curve sat clearly above the others early on: 0.2330 at 10k and 0.2326 at 20k, while every other run was already near 0.21. It converged by 100k, but it is the sole configuration with no regularization of any kind — no latent, no KL, no noise token — and it is also the one given the most capacity, three ResNet-18 backbones and 903 encoder tokens.

A plausible reading is that the CVAE's contribution here is capacity control rather than mode-encoding (the β sweep having ruled out the latter), and that it only becomes visible once the model has enough capacity to misuse. Two runs is not evidence. Offered as a hypothesis for the rollouts to test, not a conclusion.

✅ Camera keys are correct — and all THREE are required

This policy will not run without the overhead camera connected. Its two-camera sibling (..._chunk50_bc) takes wrist+front only.

Keys verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the transposed mapping from the 8bin models.

Held-out episodes — evaluate on these

[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]

5 episodes from each of the 6 object x container blocks. By episode_index: red cube (25 mm) 0-39, yellow cylinder 40-79, white cube (45 mm) 80-119; cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin is always the target.

Inputs / outputs

903 encoder tokens (301 per camera at 640x480, plus latent and state), against 602 for the two-camera variants. Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights — load via ACTPolicy.from_pretrained.

🚨 All six variants are indistinguishable on held-out loss

The full spread is 0.0079. A single run's eval_loss bounces by ~0.008 between consecutive checkpoints. The between-model differences do not exceed the within-model noise.

And the metric is biased for this data: the demonstrations contain roughly three distinct grasp approaches per object, so held-out L1 rewards a policy that averages them over one that commits to a single valid approach. A worse number here may be a better policy on the robot.

Only scored rollouts can rank these.

Training

eval_loss is flat from step 60,000 to 100,000, so no overfitting penalty; this is the 100,000-step checkpoint.

Known limitations

• n_obs_steps=1, so no velocity information.
• Scene-dependent: the cardboard box and white bin as used during recording, and the camera poses as mounted. A moved camera or a different container is a domain shift.
• One room, one lighting setup, one operator.
• ⚠️ A camera frame-rate warning was observed on battery power during roughly the first 50 episodes of collection. Reported but unconfirmed — a pixel-level check found 4-9% stalled frames uniformly with no early-vs-late pattern.
• ⚠️ The white cube is low contrast against the light table, most visibly in the overhead view — which this policy uses. Episodes 80-119 are the likeliest to underperform.