nano-vla-reach

physicalai-bmi/nano-vla-reach

A tiny open-vocabulary vision-language-action (VLA) policy that runs fully in your browser — no server, no GPU, no install. It looks at a scene, reads a free-form instruction you type, and acts. Run it live at https://physicalai-bmi.org/research/vla.

A VLA maps what it sees + what it's told → what it does. Frontier VLAs (π0, GR00T, SmolVLA) are hundreds of millions of parameters and need a GPU. This policy is 63,026 parameters; its whole forward pass — convolutions included — runs in plain JavaScript. The language is open-vocabulary: instructions are embedded on-device by all-MiniLM-L6-v2 (the same encoder at train and inference), so paraphrases it never saw still work. A faithful miniature of the paradigm, meant to be understood and run.

What it does

• Vision: a 24×24×3 image of a scene with a red target and a green target at random positions, plus the end-effector.
• Language (open-vocab): any instruction → a 384-d mean-pooled, normalized embedding from all-MiniLM-L6-v2. "reach the red target", "grab the green one", "head toward crimson", "I want green" all work.
• Action: a 2-D step for the end-effector.

The task is built so the policy must use both modalities: targets move every scene (it has to see them) and either can be the goal (it has to read the instruction).

Architecture

Conv(3→8, s2) → Conv(8→16, s2) → flatten(576) → concat[ text embedding(384) ] → MLP(→64→2), SiLU. 63,026 parameters. The 384-d language vector comes from all-MiniLM-L6-v2, run identically in Node (training) and transformers.js (browser) for exact parity.

Results (honest, closed-loop, held-out scenes)

The novel-instruction row is the open-vocab result: unseen phrasings still reach the right target. The flipped row proves it truly obeys language — with the wrong instruction it goes to the other target. Small reference policy on a synthetic task, not a foundation model, not safety-tested for deployment.

Files

• model.safetensors — weights
• vla.web.json — the same weights as float32 arrays for in-browser inference (conv+MLP, verified bit-for-bit vs safetensors; max abs diff 7×10⁻⁸)
• metrics.json, inference.py

Language embedding (must match at train + inference)

from sentence_transformers import SentenceTransformer
enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
lang = enc.encode("grab the green one", normalize_embeddings=True)  # (384,)
# then act(img, lang) — see inference.py

In the browser we use transformers.js Xenova/all-MiniLM-L6-v2 (pooling:'mean', normalize:true, dtype q8).

Released under CC-BY-4.0 by the Institute for Physical AI @ BMI. See also our first released checkpoint, forge-arm-reach-bc.