nano-vla-stack

physicalai-bmi/nano-vla-stack

A multi-step manipulation vision-language-action policy that runs fully in your browser. Instruction: "stack the red on the green." The arm grasps the named source block, carries it, and places it on the target block — a real two-phase task, not a single reach. Run it live at https://physicalai-bmi.org/research/vla.

• Vision: a 32×32 render of a 2-link arm + a red and a green block. When a block is grasped it is drawn at the gripper, so the phase (reaching vs carrying) is visible to the policy.
• Language: the source block is parsed from the instruction (first colour mentioned = source; open-vocab colour grounding) → a 2-way source one-hot. This is deliberate: mean-pooled sentence embeddings are order-blind ("red on green" vs "green on red" classifies at ~0.67), so we parse the order and let the policy condition on the resolved source/target.
• Action: two joint deltas. Grasp/place are automatic on proximity; the policy drives the arm.
• Params: 55,666.

Results (held-out, closed-loop, full episode)

The flipped row shows the language matters: tell it to move the other block and it does, so success measured against the original target drops sharply. Trained by behavior cloning on a Jacobian-IK two-phase expert (100% ceiling) with DART state-noise injection to survive closed-loop drift across both phases.

Files: model.safetensors, vla.web.json (float32 for in-browser; forward verified 7.6e-8 vs safetensors), metrics.json. CC-BY-4.0, Institute for Physical AI @ BMI. See also nano-vla-arm-3d (perspective reach) and nano-vla-reach.