nano-vla-arm

physicalai-bmi/nano-vla-arm

An open-vocabulary vision-language-action policy that drives an articulated arm, running fully in your browser — no server, no GPU, no install. It looks at a rendered camera view of a 2-link arm and two colored blocks, reads an instruction you type in your own words, and outputs joint commands to move the arm to the block you named. Run it live at https://physicalai-bmi.org/research/vla.

A VLA maps what it sees + what it's told → what it does. This one is 92,338 parameters and runs its whole forward pass — three convolution layers and all — in plain JavaScript on your device. The language is real and open-vocabulary: the instruction is embedded on-device by all-MiniLM-L6-v2 (the same encoder used at training time), so phrasings it never saw still work. It is a faithful miniature of the paradigm, meant to be understood and run — not a foundation model.

What it does

• Vision: a 32×32×3 rendered camera view of a 2-link arm (base, links, gold gripper) and a red and green block at random positions.
• Language (open-vocab): any instruction — "reach the red target", "grab the green one", "head toward crimson" — embedded to 384-d on your device.
• Action: two joint deltas (Δθ₁, Δθ₂). The arm is stepped forward; the loop closes in the browser.

The task is built so the policy must use all three modalities: the blocks and the arm move every scene (it has to see), either block may be the goal (it has to read the instruction), and it controls the arm through its joints (real articulated action, not a teleport).

Architecture

Conv(3→8, s2) → Conv(8→16, s2) → Conv(16→32, s2) → flatten → concat[ MiniLM text embedding, 384-d ] → MLP(→96→2), SiLU. 92,338 parameters.

Results (honest)

Closed-loop on held-out scenes, driving the arm joint-by-joint:

The novel-instruction row is the open-vocab proof: unseen phrasings still work, because the on-device text encoder maps them near the ones it learned. The flipped row is the language-grounding proof: with the wrong instruction it drives to the other block, so "correct" success collapses to zero.

Trained by behavior cloning on a scripted Jacobian-IK expert, made robust to closed-loop drift with DART-style state-noise injection (the policy sees off-expert arm configurations + the expert's recovery there — the same idea as DAgger). Small synthetic task; not a foundation model, not safety-tested for any hardware.

Files

• model.safetensors — weights
• vla.web.json — the same weights as float32 arrays for in-browser inference (conv + MLP; verified bit-for-bit equal to safetensors; max abs diff 1.7×10⁻⁷)
• metrics.json — the numbers above
• inference.py — minimal NumPy example (renderer + forward pass)

Language encoder

Instructions are embedded with Xenova/all-MiniLM-L6-v2 (q8) via Transformers.js on-device. The policy conditions on the 384-d sentence embedding, so the vocabulary is open — paraphrases and common color synonyms generalize.

Citation

@misc{nano_vla_arm_2026,
  title  = {nano-vla-arm: an open-vocab in-browser vision-language-action arm policy},
  author = {Institute for Physical AI at Bailey Military Institute},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/physicalai-bmi/nano-vla-arm}}
}

Released under CC-BY-4.0 by the Institute for Physical AI @ BMI. See also nano-vla-reach (the end-effector predecessor) and forge-arm-reach-bc (our first checkpoint).