RPT-VLA

RPT-VLA

This repository contains a Hugging Face-style export of RPT-VLA, a diffusion-action vision-language-action model for SimplerEnv evaluation. The model takes a language instruction and an RGB observation as input, then predicts 7-DoF robot actions of the form (x, y, z, roll, pitch, yaw, gripper).

The checkpoint is packaged for offline local loading: model weights, tokenizer files, configuration files, and dataset statistics are stored in a single top-level directory.

Model Summary

• Model type: Vision-language-action policy with a diffusion action decoder
• Base VLM: prism-dinosiglip-224px+7b
• Vision backbone: DINOv2 + SigLIP fused vision backbone
• Language model: Llama-2 7B backbone
• Action decoder: DiT-B diffusion action head
• Action dimension: 7
• Future action window size: 15
• Past action window size: 6
• Repeated diffusion steps: 4
• Supported normalization keys: fractal20220817_data, bridge_dataset
• Checkpoint format: flat Hugging Face-style safetensors shards

Files

• config.json: model configuration used by the local diffusion-action loader
• config.yaml: original run configuration in YAML form
• dataset_statistics.json: action normalization statistics for supported evaluation datasets
• model.safetensors.index.json: index mapping parameters to safetensors shards
• model-*.safetensors: sharded model weights
• tokenizer.model, tokenizer.json, tokenizer_config.json, special_tokens_map.json, added_tokens.json: tokenizer files
• llama2_7b_config.json: local Llama-2 backbone configuration used for offline model construction
• offline_backbones.py: offline construction helper for local evaluation environments
• model_format.json: lightweight metadata describing the exported checkpoint layout

Uses

This model is intended for SimplerEnv robot policy evaluation and related fine-tuning workflows. It can be evaluated on task families whose action normalization statistics are included in dataset_statistics.json.

For execution, actions must be un-normalized with the correct unnorm_key:

• fractal20220817_data for Google Robot-style tasks
• bridge_dataset for WidowX / Bridge-style tasks

Using the wrong normalization key can produce invalid actions and unreliable evaluation results.

Getting Started

Use the accompanying project evaluation scripts and pass this directory as --model-dir.

Example for Google Robot-style tasks:

python scripts/eval.py \
  --robot google_robot \
  --model-dir /path/to/model \
  --simpler-env-root /path/to/simpler_env \
  --log-root /path/to/output/google_robot \
  --gpu 0 \
  --families ALL \
  --protocol both

Example for WidowX / Bridge-style tasks:

python scripts/eval.py \
  --robot widowx_robot \
  --model-dir /path/to/model \
  --simpler-env-root /path/to/simpler_env \
  --log-root /path/to/output/widowx_robot \
  --gpu 0

Evaluation Arguments

The evaluation entry is scripts/eval.py. The main configurable arguments are:

Google Robot supports these task families:

• pick_coke_can
• move_near
• drawer
• put_in_drawer

WidowX evaluation currently runs the bundled Bridge task set and does not use --families or --protocol.

Each evaluation output directory contains:

• progress.jsonl: rollout-level results
• progress_summary.txt: aggregate success rates
• videos/: rollout videos and action visualizations, if enabled

Limitations

The model is designed for the robot embodiments and evaluation setups represented by its normalization statistics and training data. It should not be assumed to transfer directly to unseen robot embodiments or substantially different camera/action conventions without additional validation or fine-tuning.