nano-vla-pixels-dart

physicalai-bmi/nano-vla-pixels-dart

A pixel-only control policy for an in-browser MuJoCo arm, trained on frames captured through the same in-page WebGPU readback path it runs on — so train and live are one distribution by construction. Runs live at /research/vla.

• Input: 3 stacked 48x48 RGB frames (newest first, 9ch) — no joint angles, no coordinates.
• Output: 3 joint deltas. 186,099 params, plain-JS forward, ~9 ms read+infer.
• Data: readback frames = DART (expert perturbed so the arm visits off-distribution states; the expert's clean action is the label) + one DAgger round with task variety forced during capture.

Result

Driving from pixels alone at ~30 Hz it completes ~23 reaches/min — the state-based expert checkpoint it was cloned from does ~21. It matches its teacher from the rendered picture alone.

The correction this artifact exists to record

For most of its development this policy scored ~1 reach/min and looked like a near-failure. The cause was not the policy — it was the control rate. The demo re-perceived every 90 ms (an arbitrary constant), so the policy decided at ~11 Hz while the expert re-decided every frame at ~60; its action went stale and the arm overshot. Dropping the interval to 30 ms (still ~9 ms of real work) took the rate from ~1 to ~23/min.

That one number had silently poisoned every comparison run through it. Measured at 11 Hz, different training runs scored 0–1/min and we built causal stories on the noise ("DAgger collapse", "v5 is the peak"). Re-measured at 30 Hz they are indistinguishable — v5, v6 (which we had called "collapsed"), and v8 all land ~22–23/min. What genuinely mattered: the readback train==live capture + DART lifted a screenshot-trained baseline from ~10/min to parity. The extra DAgger rounds we had credited moved nothing.

Method lesson: held-out MSE and expert-agreement were blind to all of this — they barely moved while the true rate went 0 → 23. Only end-to-end rollouts, at a sane control rate, rank these models.

Released CC-BY-4.0 by the Institute for Physical AI @ BMI.