tt-openvla

tt-openvla

A TT-Metal / TTNN bring-up of OpenVLA-7B, a vision-language-action model for robot manipulation.

• What it does: given a camera image and a natural-language instruction, it predicts a 7-DoF end-effector action (dx, dy, dz, droll, dpitch, dyaw, gripper).
• How: a fused DINOv2 + SigLIP vision backbone feeds a LLaMA-2-7B language model. The action comes out as extra vocabulary tokens and is decoded back through a bin mapping.
• Hardware: a 2-chip Blackhole P300 mesh.
• Status: Experimental.
  • On 2 of 4 test inputs, the action tokens match the Hugging Face reference exactly.
  • On the other two, the largest action difference is 0.045, from bf16 numerical drift.
  • No task-success evaluation has been run (see Limitations).
• Code: tsingletaryTT/tt-openvla.

Packaged and published with tt-model-manager as a v6 thin bundle (manifest schema 6): a pip/venv install, not a container.

No checkpoint is hosted here. Weights are fetched at install or serve time from openvla/openvla-7b at a pinned revision, unconverted. The two vision towers, the projector and LLaMA-2-7B all load from that checkpoint's own tensors.

At a glance

Intended use

Direct use: research and bring-up evaluation of OpenVLA inference on Tenstorrent hardware:

• interactive exploration through the Gradio UI;
• offline or simulated action prediction through POST /act, for comparison against GPU OpenVLA.

Out-of-scope use:

• Driving a physical robot without an independent safety layer, meaning workspace limits, velocity and force clamps, an e-stop and a human supervisor. This port hasn't been evaluated on any robot or simulator task, and its actions aren't bit-identical to the reference model (see Limitations).
• Robot embodiments or scenes outside the Open X-Embodiment pretraining mix. Upstream states that OpenVLA doesn't generalize zero-shot to unseen embodiments or setups, and recommends fine-tuning instead.
• Any use the Llama 2 Acceptable Use Policy prohibits. That includes the §2 activities that present a risk of death or bodily harm, for example "Operation of critical infrastructure, transportation technologies, or heavy machinery".
• Safety-critical, medical, or unsupervised deployment of any kind.

Quickstart

uv tool install tenstorrent   # once: the Tenstorrent CLI, `tt`
tt model pull episod/tt-openvla
tt serve episod/tt-openvla

tt model pull installs the v6 thin bundle: a hermetic uv venv with a pinned Python 3.12, ttnn==0.77.0, and the two bundled wheels.

• Weights: it also fetches openvla/openvla-7b at revision 47a0ec7 (about 15 GB) into the bundle's own HF cache (<install>/.hf). Weights come down by default; tt has no --with-weights flag.
• Port: serve starts uvicorn on port 20000, or the next free port. The Gradio UI is at / and the REST endpoint at /act.
• Readiness: the mesh open and model load run in the FastAPI lifespan, so uvicorn's Application startup complete. means the backend is ready.

Measured on a fresh install (one P300c board, 2026-09-27):

No other model repos are fetched, apart from a small NousResearch LLaMA-2 config/tokenizer that tt-metal's tt_transformers reads.

Without tt-cli, tt-model alone does the whole job:

tt-model pull  episod/tt-openvla --with-weights
tt-model serve episod/tt-openvla

Serve profiles

One prediction runs at a time. The Gradio UI is limited to concurrency_limit=1. /act requests and UI requests share one lock around the backend, so concurrent requests queue rather than run two decode loops on the mesh.

Using it

Not an OpenAI-compatible API. A continuous action vector isn't natural-language output, so there is no /v1/chat/completions.

Gradio UI (/)

1. Upload an image, or use the bundled example.
2. Write an instruction.
3. Choose the unnormalization dataset (default bridge_orig).
4. Click Predict action.

The UI shows the decoded 7-DoF action, the raw generated action-token IDs, and the measured latency.

REST: POST /act

The request is the one openvla/openvla's reference server (vla-scripts/deploy.py) takes:

• Plain form: a json-numpy-encoded {image, instruction, unnorm_key?}.
• Encoded form: {"encoded": ...}.

Both build the same prompt as upstream: In: What action should the robot take to {instruction.lower()}?\nOut:, plus the empty token upstream's predict_action appends.

It differs from upstream in three ways:

• Response shape: {"action": [...]}, where upstream returns the bare array. Upstream clients that do action = resp.json() need resp.json()["action"].
• Default unnorm_key: bridge_orig. Upstream defaults to None, which errors for openvla-7b.
• Errors: a malformed payload, including a non-string instruction, returns HTTP 400 with a detail message. Upstream returns 200 with an "error" field.

import json_numpy; json_numpy.patch()  # pip install json-numpy
import requests, numpy as np
from PIL import Image

image = np.array(Image.open("my_photo.jpg").convert("RGB"))  # uint8 HxWx3
resp = requests.post("http://localhost:20000/act", json={
    "image": image,
    "instruction": "pick up the remote control",
    # "unnorm_key": "bridge_orig",   # optional; any key in openvla-7b config.json norm_stats
})
action = resp.json()["action"]  # 7 floats: dx, dy, dz, droll, dpitch, dyaw, gripper

• Shared backend: the route reuses the backend the UI loaded, so there's no second model load.
• Determinism: decoding is greedy. Repeated and concurrent requests for the same input returned identical actions.

From the GitHub repo

• gradio_app/app.py runs the same demo directly: on Blackhole by default (hold a 2-chip lease first), or on CPU with --backend reference.
• tt/hf_reference.py runs the upstream HF AutoModelForVision2Seq reference that the accuracy numbers below were measured against.

Expected performance

Methodology: the latency and concurrency figures come from HTTP POST /act on one P300c board (2 Blackhole chips, 1×2 mesh, ttnn==0.77.0 from PyPI). The request was the bundled gradio_app/assets/example.jpg with "open the drawer" and unnorm_key=bridge_orig, json-numpy encoded, measured client-side. There was one untimed first call, then N=20 sequential warm calls and one 4-way concurrent burst. The run used this bundle revision (weights 47a0ec7) on 2026-09-27.

The accuracy rows come from a separate run on the same board against the upstream HF reference on CPU (timm 0.9.16, transformers 4.40.1, greedy). It used four image/instruction pairs: the repo's example image with "open the drawer", a house in a field with "move forward", a dog with "Pick up the ball", and a synthetic orange image with "move forward". The script is tt/hf_reference.py in the GitHub repo.

Why the remaining token mismatches look like precision, not a bug:

• Per layer: all-token hidden-state PCC against HF is at least 0.99995 at every layer. Last-token PCC decays smoothly, from 0.9999 at layer 0 to about 0.997 at layer 30, with no step change.
• Where the flips happen: at low-margin steps. HF's own bf16 run flips a token on one of the same inputs.
• Not tried yet: precision options such as an fp32 LM head or HiFi4 decode.

Limitations

• Actions are close to the reference, not identical to it.
  • Two of the four test inputs match HF token for token. On the other two, the first divergent token appears at a low-margin step, and the decoded action differs by at most 0.045.
  • Outputs are deterministic run to run, but a different input can land on a different bin than GPU OpenVLA.
• No task-level evaluation. There are no BridgeData V2, LIBERO, or real-robot results. The hardware checks used four still images, not robot scenes or closed-loop control.
• The server binds 0.0.0.0 with no authentication.
• One validated topology: P300, 2 chips. A single chip doesn't fit (L1 overflow in decode).
• /act is serialized. Throughput does not scale with concurrent clients: four simultaneous requests finish one after another.
• Chatty logs. tt-metal logs a Failed to discover available ethernet links warning hundreds of times per /act call. It is harmless, but it makes the log grow fast.

Risks and safety considerations

• Physical harm. The model outputs end-effector deltas and a gripper command. If a robot executes them without independent limits, wrong actions can:

  • move it unexpectedly;
  • collide with people or objects;
  • drop or crush what it's holding.

  Nothing in this package bounds, filters, or sanity-checks actions. Keep a safety layer and an e-stop in the loop, and prefer simulation.

• Silent numerical deviation. On some inputs this port produces a different action than the reference (see Limitations), and the output looks plausible either way. A wrong action and a right one are indistinguishable from the response alone.

• Out-of-distribution inputs. For scenes, cameras, or embodiments outside the training mix, OpenVLA still returns a confident-looking action rather than refusing.

• Unnormalization choice. Actions are unnormalized with per-dataset statistics (unnorm_key). The wrong key for your robot rescales every motion, and the server defaults silently to bridge_orig.

• Network exposure. An unauthenticated /act on 0.0.0.0 lets anyone on the network issue actions to a connected control loop. Bind to localhost, or put it behind a proxy with auth.

• Supply chain. The processor loads with trust_remote_code=True, which executes the upstream repo's Python at serve time. That code is pinned to revision 47a0ec7, the same revision as the weights.

• Pass these risks on. The Llama 2 AUP (§4) requires disclosing known dangers of your AI system to end users. If you build on this package, pass these risks along.

Licensing

This repo hosts no weights. Downloading and running openvla-7b is between you and its licensors.

Changelog

The 2026-09-27 release (tt-openvla-serving 0.2.1):

• Vision towers: now loaded from openvla-7b's own checkpoint tensors, replacing timm's generic pretrained weights.
• Fresh installs: weights are downloaded at the pinned revision 47a0ec7. This fixes the "expected 3 safetensors shards, found []" startup failure.
• Two RoPE bugs fixed, which had masked each other:
  • a missing RoPE initialization in the CPU reference;
  • the wrong rotation pairs in TTNN.
• Prompt: the empty token upstream's predict_action appends is now added.
• /act:
  • serialized, so a 4-way concurrent burst no longer hangs the server;
  • lowercases the instruction;
  • accepts {"encoded": ...};
  • returns 400 on a bad payload.
• Manifest: records the weights revision and tt_metal_version 0.77.0.
• Re-verified on hardware from a fresh install with an empty HF cache: 334 ms warm /act, 4/4 concurrent requests OK.

Related packages

None.

Feedback

• Questions or problems with this package: open a discussion at https://huggingface.co/episod/tt-openvla/discussions. That is the one channel that reaches the bundle's author.
• A problem with the tt tooling itself: use tt report issue. It files against tenstorrent/tt-cli and doesn't reach this package's author.
• Product feedback: [email protected].

Provenance