openvla-oft-dexverse

OpenVLA-OFT DexVerse baseline checkpoints

OpenVLA-OFT policies finetuned on the DexVerse tabletop dexterous-manipulation benchmark, for use with the online evaluation harness in scripts/eval/.

Two embodiments of a floating Shadow hand are covered:

State is a single (non-stacked) frame — openvla-oft's RLDS transform clips proprio to one frame. eval_policy.py --policy openvla truncates to 28/56 accordingly and applies no observation preset.

Download

# single-hand
hf download dexverse/openvla-oft-dexverse --include 'single/*' --local-dir ./openvla-ckpt
# bimanual
hf download dexverse/openvla-oft-dexverse --include 'bimanual/*' --local-dir ./openvla-ckpt

CKPT_DIR for serve_openvla.sh is then ./openvla-ckpt/single (or bimanual).

⚠️ unnorm_key must contain dexbench

openvla-oft selects its action and proprio dimensions by inspecting sys.argv at import time (prismatic/vla/constants.py::detect_robot_platform). It scans the joined argv for dexbench, libero, aloha, bridge in that order and silently falls back to LIBERO's 7-dim constants when nothing matches. The only argv token carrying that keyword is --unnorm_key.

So the key must be dexbench_rlds/single / dexbench_rlds/bimanual exactly — these are also the only keys present in dataset_statistics.json. dexbench is the benchmark's former name; renaming it here requires a matching change in openvla-oft. serve_openvla.sh refuses to start when the key lacks the keyword.

Contents

single/
├── model-0000{1,2,3,4}-of-00004.safetensors   # merged 7B VLA backbone, ~15 GB
├── model.safetensors.index.json
├── action_head--3000_checkpoint.pt            # L1-regression action head, 1.0 GB
├── proprio_projector--3000_checkpoint.pt      # proprio encoder, 68 MB
├── dataset_statistics.json                    # action/proprio denorm stats
├── lora_adapter/                              # provenance only, see below
├── config.json, configuration_prismatic.py, modeling_prismatic.py,
│   processing_prismatic.py, preprocessor_config.json, processor_config.json
└── tokenizer.json, tokenizer.model, tokenizer_config.json, ...

The LoRA adapter is already merged into the sharded safetensors that deploy.py loads. lora_adapter/ is kept for provenance and is not needed to serve the model.

Serving

The policy runs out of process — its torch/transformers pins are not compatible with Isaac Sim's. From a DexVerse checkout:

OPENVLA_OFT_ROOT=/path/to/openvla-oft \
PYTHON_BIN=/path/to/envs/openvla-oft/bin/python \
CKPT_DIR=$PWD/openvla-ckpt/single \
UNNORM_KEY=dexbench_rlds/single \
    bash scripts/eval/serve_openvla.sh          # add EMBODIMENT=bimanual for the pair

# in the DexVerse / Isaac Lab environment
python scripts/eval/eval_policy.py --policy openvla --enable_cameras --headless \
    --unnorm_key dexbench_rlds/single --task Dexverse-GraspCup-v0 --num_episodes 20

EMBODIMENT=bimanual switches --num_images_in_input from 2 to 3. The server was trained with --use_l1_regression True --use_proprio True --center_crop True and LoRA rank 32; serve_openvla.sh passes all of these.

The wire format the server expects — produced by eval_policy.py:

POST /act  (json_numpy body)
{"full_image": uint8 (256,256,3),
 "wrist_image": uint8 (256,256,3),   # left_wrist_image + right_wrist_image when bimanual
 "state": float32 (28,),             # (56,) when bimanual
 "instruction": str,
 "unnorm_key": "dexbench_rlds/single"}   # -> list of 8 actions, already denormalized

The instruction strings are part of the benchmark definition and are listed in scripts/eval/baseline_tasks.py; they are the RLDS language_instruction values verbatim.

Training

LoRA-finetuned (rank 32) from openvla/openvla-7b for 3000 steps on demonstrations recorded through the DexVerse VR teleoperation pipeline, converted to RLDS. Backgrounds and table textures were held fixed during data generation (create_demo_files_sequential.py --disable-bg-randomize); evaluate with the matching setting (the harness default) or expect a visual domain gap.

License

MIT, inherited from the openvla/openvla-7b base model. The DexVerse code is BSD-3-Clause.