vlajepa-rlbench16

Model Card for vla_jepa

<!-- Provide a quick summary of what the model is/does. -->

VLA-JEPA is a Vision-Language-Action model that combines a Qwen3-VL language backbone with a self-supervised video world model (V-JEPA2) and a flow-matching DiT action head.

<!-- A short demo is worth more than any description! Record a GIF/video of the policy running on your robot, upload it to this repo, and embed it here: <p align="center"> <img src="https://huggingface.co/<hf_user>/<policy_repo_id>/resolve/main/demo.gif" width="60%"/> </p> -->

This policy has been trained and pushed to the Hub using LeRobot.

Learn how to train and run it in the LeRobot vla_jepa guide, or browse the full documentation.

---

Model Details

• License: apache-2.0
• Cameras: exterior_1_left, exterior_2_left

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Outputs

Training Dataset

• Repository: local/rlbench16
• Episodes: 4200
• Frames: 741312
• Frame rate: 20 FPS
• Task(s): "pick up the lid from the table and put it on the red jar", "grasping the lid, lift it from the table and use it to seal the red jar", "screw on the red jar lid", "close the red jar", "screw on the maroon jar lid", "close the maroon jar", "pick up the lid from the table and put it on the maroon jar", "pick up the lid from the table and put it on the lime jar", "grasping the lid, lift it from the table and use it to seal the lime jar", "close the lime jar", "screw on the lime jar lid", "slide the ring onto the red colored spoke", "place the ring onto the red spoke", "put the ring on the red spoke", "place the ring onto the maroon spoke", "put the ring on the maroon spoke", "slide the ring onto the maroon colored spoke", "place the ring onto the lime spoke", "slide the ring onto the lime colored spoke", "put the ring on the lime spoke", "screw the light bulb from the red holder into the lamp", "pick up the light bulb from the red stand, lift it up to just above the lamp, then screw it down into the lamp in a clockwise fashion", "put the light bulb from the red casing into the lamp", "screw in the red light bulb", "put the light bulb from the maroon casing into the lamp", "screw in the maroon light bulb", "pick up the light bulb from the maroon stand, lift it up to just above the lamp, then screw it down into the lamp in a clockwise fashion", "screw the light bulb from the maroon holder into the lamp", "screw the light bulb from the lime holder into the lamp", "put the light bulb from the lime casing into the lamp", "screw in the lime light bulb", "take the chicken off the grill", "pick up the chicken and place it next to the grill", "remove the chicken from the grill and set it down to the side", "pick up the steak and place it next to the grill", "take the steak off the grill", "remove the steak from the grill and set it down to the side", "open bottom drawer", "grip the bottom handle and pull the bottom drawer open", "slide the bottom drawer open", "grip the middle handle and pull the middle drawer open", "slide the middle drawer open", "open middle drawer", "open top drawer", "grip the top handle and pull the top drawer open", "slide the top drawer open", "pick up the cube and put it in the sorter", "slot the cube into the shape sorter", "put the cube in the shape sorter", "place the cube into its slot in the shape sorter", "place the cylinder into its slot in the shape sorter", "put the cylinder in the shape sorter", "pick up the cylinder and put it in the sorter", "slot the cylinder into the shape sorter", "pick up the triangular prism and put it in the sorter", "slot the triangular prism into the shape sorter", "place the triangular prism into its slot in the shape sorter", "put the triangular prism in the shape sorter", "press the maroon button", "push down the button with the maroon base", "push the maroon button", "push the maroon button, then push the green button", "push down the button with the maroon base, then the green one", "press the maroon button, then press the green button", "push down the button with the maroon base, then the green one, then the blue one", "press the maroon button, then press the green button, then press the blue button", "push the maroon button, then push the green button, then push the blue button", "put the crackers in the cupboard", "move the crackers to the bottom shelf", "pick up the crackers and place it in the cupboard", "put away the crackers in the cupboard", "put away the chocolate jello in the cupboard", "put the chocolate jello in the cupboard", "pick up the chocolate jello and place it in the cupboard", "put away the strawberry jello in the cupboard", "pick up the strawberry jello and place it in the cupboard", "move the strawberry jello to the bottom shelf", "put the strawberry jello in the cupboard", "open the bottom drawer and place the block inside of it", "leave the block in the bottom drawer", "put the block away in the bottom drawer", "put item in bottom drawer", "open the middle drawer and place the block inside of it", "put item in middle drawer", "put the block away in the middle drawer", "open the top drawer and place the block inside of it", "leave the block in the top drawer", "put the block away in the top drawer", "put item in top drawer", "leave the money on the bottom shelf on the safe", "put the money away in the safe on the bottom shelf", "place the stack of bank notes on the bottom shelf of the safe", "put the money away in the safe on the middle shelf", "leave the money on the middle shelf on the safe", "place the stack of bank notes on the middle shelf of the safe", "leave the money on the top shelf on the safe", "put the money away in the safe on the top shelf", "place the stack of bank notes on the top shelf of the safe", "use the stick to drag the cube onto the red target", "drag the block towards the red square on the table top", "grasping the stick by one end, pick it up and use the its other end to move the block onto the red target", "pick up the stick and use it to push or pull the cube onto the red target", "drag the block towards the maroon square on the table top", "pick up the stick and use it to push or pull the cube onto the maroon target", "grasping the stick by one end, pick it up and use the its other end to move the block onto the maroon target", "use the stick to drag the cube onto the maroon target", "grasping the stick by one end, pick it up and use the its other end to move the block onto the lime target", "drag the block towards the lime square on the table top", "use the stick to drag the cube onto the lime target", "slide the block towards the green target", "slide the block onto the target", "push the block until it is sitting on top of the target", "cover the target with the block by pushing the block in its direction", "slide the block to target", "stack 2 red blocks", "arrange 2 red blocks in a vertical stack on the table top", "set 2 red cubes on top of each other", "pick up and set down 2 red blocks on top of each other", "build a tall tower out of 2 red cubes", "place 2 of the red cubes on top of each other", "pick up and set down 3 red blocks on top of each other", "set 3 red cubes on top of each other", "arrange 3 red blocks in a vertical stack on the table top", "place 3 of the red cubes on top of each other", "stack 3 red blocks", "build a tall tower out of 3 red cubes", "set 4 red cubes on top of each other", "place 4 of the red cubes on top of each other", "arrange 4 red blocks in a vertical stack on the table top", "build a tall tower out of 4 red cubes", "pick up and set down 4 red blocks on top of each other", "place the wine bottle on the wine rack", "stack wine bottle", "leave the wine on the shelf", "slide the bottle onto the wine rack", "grasp the bottle and put it away", "put the wine away", "leave the table clean", "sweep the dirt up", "use the broom to brush the dirt into the dustpan", "clean up the dirt", "pick up the brush and clean up the table", "sweep dirt to dustpan", "grasping the broom by its handle, clear way the dirt from the table", "grasp the left tap and turn it", "turn left tap", "rotate the left tap", "rotate the right tap", "turn right tap", "grasp the right tap and turn it"

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=local/rlbench16"> <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/> <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/> </a>

Training Configuration

---

How to Get Started with the Model

New to LeRobot? These guides cover the full workflow:

• Install LeRobot — set up the lerobot package.
• Hardware setup — assemble, wire, and calibrate your robot and cameras.
• Record data & train a policy — the end-to-end imitation-learning walkthrough.
• CLI cheat-sheet — quick reference for the lerobot-* commands.

The short version to run and train this policy:

Run the policy on your robot

lerobot-rollout \
  --strategy.type=base \
  --robot.type=<your_robot_type> \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ <camera_1>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}, <camera_2>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}}" \
  --policy.path=TingtingDu/vlajepa-rlbench16 \
  --task="pick up the lid from the table and put it on the red jar" \
  --duration=60

Replace the remaining <...> placeholders with your own values: --robot.port and the camera names/indices are specific to your machine, and the camera names must match the observation keys this policy was trained on.

When --strategy.type=base is used the script doesn't record the episodes. Skipping duration will make the policy run indefinitely. For more information look at rollout documentation.

Train your own policy

lerobot-train \
  --dataset.repo_id=${HF_USER}/<dataset> \
  --policy.type=vla_jepa \
  --output_dir=outputs/train/<policy_repo_id> \
  --job_name=lerobot_training \
  --policy.device=cuda \
  --policy.repo_id=${HF_USER}/<policy_repo_id> \
  --wandb.enable=true

Writes checkpoints to outputs/train/<policy_repo_id>/checkpoints/.

---

Evaluation

<!-- Report real-robot results here: run the policy several times per task and count the successes. Delete the "No evaluation results" line and fill in this table instead: | Task | Trials | Successes | Success rate | | ---- | ------ | --------- | ------------ | | pick the lego brick | 10 | 8 | 80% | Also worth noting: anything that affects difficulty (new object positions, lighting, distractors, a different robot of the same type, ...). -->

No evaluation results have been provided for this policy yet.

---

Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}