GEAR-VLA

yuanzhang/GEAR-VLA

GEAR-VLA is a machine learning model from yuanzhang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

<p align="center" ⭐️ <a href="https://babynabeauty.github.io/gear-vla-p/"<bProject Page</b</a   |   📖 <a href="https://arxiv.org/pdf/2606.08530"<bPaper</b</a   |   🗂️ <a href=…

Downloads · 30 days

23% of all-time downloads

All-time downloads

Public

Parameters

8.9B

17.7 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors17.7 GB · 100%

At a glance

License: mit
Model type: internvl_vggt_chat
Access: Public
Created: Aug 2, 2026
Updated: Aug 14, 2026
SHA: fef3809c

Model card

From the Hugging Face model README

GEAR-VLA-9B

⭐️ <a href="https://babynabeauty.github.io/gear-vla-p/">Project Page</a>   |   📖 <a href="https://arxiv.org/pdf/2606.08530">Paper</a>   |   🗂️ <a href="https://huggingface.co/datasets/yuanzhang/GEAR-VLA-Dataset">Dataset</a> <img src="https://huggingface.co/yuanzhang/GEAR-VLA/resolve/main/images/gear-vla-overview.png" style="width: 100%; height: auto; object-fit: contain;" alt="GEAR-VLA framework overview" >

🔥 Introduction

GEAR-VLA is a vision-language-action framework designed to connect embodied perception, geometric reasoning, and robot control. This repository releases GEAR-VLA-9B, the vision-language model (VLM) checkpoint obtained after the Embodied Pretraining stage.

GEAR-VLA introduces three complementary designs:

🎯 Coarse-to-Fine Action Learning. The framework combines latent action representations from manipulation videos with discretized robot actions. This design first captures high-level interaction dynamics and subsequently learns precise embodiment-specific control.

🧭 Semantic-Aligned 3D Geometry Integration. A trainable VGGT branch extracts multi-view 3D geometry while preserving the semantic knowledge of the frozen 2D visual pathway. The two feature streams are aligned and fused before entering the language model.

🤖 Embodiment Canonicalization. Embodiment-aware state inputs and a unified relative end-effector action space confine robot-specific variation to the low-level interface. This design supports transfer across robot platforms without introducing robot-specific semantic prompts into the shared representation.

The released checkpoint provides the VLM foundation used by the complete GEAR-VLA system. It does not include the downstream DiT action expert, robot-state projector, or a complete continuous-control policy.

📚 Embodied Pretraining

Embodied Pretraining uses approximately 4.189 million general vision-language and embodied-perception samples, together with 23,027.6 hours of robot and egocentric manipulation data. All tasks are formulated as autoregressive token prediction within a unified training objective.

For robot trajectories with action annotations, the model predicts both FAST-style action tokens and latent action identifiers. For videos without action annotations, it learns from latent action identifiers. Pretraining was conducted for 350,000 iterations on 240 NVIDIA H200 GPUs, with a batch size of eight per GPU.

🏗️ Model Architecture

GEAR-VLA-9B uses a dual-branch visual architecture. The original 2D visual encoder remains frozen to preserve pretrained semantic representations, while VGGT serves as a trainable 3D spatial encoder for multi-view geometric reasoning. The current implementation uses the VGGT Aggregator and does not use its camera, depth, point, or tracking heads.

The 2D features H2D and 3D features H3D are concatenated along the feature dimension and passed through an expanded MLP projector. The projector weights associated with the 2D pathway are inherited from the pretrained VLM, whereas the newly introduced 3D pathway is zero-initialized. This initialization preserves the original visual-language behavior at the start of training while allowing geometry-aware features to be learned progressively.

GEAR-VLA-9B architecture

📊 VLM Capability Evaluation

We evaluated the current GEAR-VLA-9B Embodied Pretraining checkpoint under the same evaluation protocols and obtained consistent results.

The evaluation covers EgoPlan2, Where2Place, RefSpatial-bench, ShareRobot-affordance, ShareRobot-trajectory, BLINK, EmbSpatial, ERQA, SAT, and CVBench. These benchmarks primarily assess embodied reasoning, spatial understanding, grounding, affordance prediction, and trajectory reasoning.

These scores characterize the embodied capabilities of the VLM checkpoint. They should not be interpreted as robot-task success rates for the complete continuous-control policy.

🏆 Complete GEAR-VLA System Results

The complete GEAR-VLA system, which includes downstream action-policy components that are not part of this release, achieves the following results reported in the paper:

Evaluation setting	Result
LIBERO	98.7%
LIBERO-Plus	88.7%
RoboTwin, clean	91.1%
RoboTwin, randomized	89.9%
AgileX	85.9%
LDT-01	81.0%

The same VLM is used for target grounding and mask-guided reasoning. Given the first-frame target mask and current multi-view observation, the DiT-based action expert predicts the robot action for manipulating the specified object.

Across 6,360 trials involving 212 previously unseen objects, the complete system achieves a grasping success rate of 90.1%.

The training split contains 35 target objects, with 100 demonstrations collected for each object. During data collection,each scene contains the target object and 4–20 background objects sampled from the training-object pool, including objects with similar object categories. The test split contains 212 target objects that do not overlap with the training objects, covering the same object categories to evaluate generalization over object geometry rather than memorization of specific object instances.

📦 Released Checkpoint

This repository contains the VLM parameters produced by the Embodied Pretraining stage. The checkpoint is intended for embodied visual-language understanding, geometric reasoning, and initialization of downstream GEAR-VLA training.

The release does not contain the DiT action expert, robot-state projector, embodiment-specific action heads, or downstream continuous-control policy. Reproducing the complete-system robot results therefore requires the remaining policy components and downstream training procedure described in the paper.

⚠️ Limitations

The VGGT branch and multi-stage training procedure increase computational and memory requirements. The current geometry integration also relies on multi-view inputs. Single-camera settings, mobile manipulation, and scenes with severe occlusion may require additional adaptation.

🚀 Quick Start

Using 🤗 Transformers to Chat

The checkpoint uses custom Transformers code and the VGGT backbone. Install the required packages before loading the model.

pip install -U "transformers>=4.48.3" huggingface_hub accelerate
pip install git+https://github.com/facebookresearch/vggt.git

<details> <summary>Python code</summary>

import torch
from huggingface_hub import snapshot_download
from PIL import Image
from transformers import AutoModel, AutoProcessor


REPO_ID = "yuanzhang/GEAR-VLA"

# Download the complete checkpoint first because the custom processor loads
# its configuration and tokenizer from a local snapshot.
model_dir = snapshot_download(REPO_ID)

processor = AutoProcessor.from_pretrained(
    model_dir,
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    device_map="auto",
).eval()

# Supply one or more synchronized views of the same scene.
views = [
    Image.open("./view_1.jpg").convert("RGB"),
    Image.open("./view_2.jpg").convert("RGB"),
]

# The processor creates aligned inputs for both visual branches:
# pixel_values: [V, 3, 448, 448]
# vggt_images:  [1, V, 3, 518, 518]
image_inputs = processor.preprocess_images(views)
pixel_values = image_inputs["pixel_values"].to(
    device="cuda",
    dtype=torch.bfloat16,
)
vggt_images = image_inputs["vggt_images"].to(
    device="cuda",
    dtype=torch.bfloat16,
)

question = (
    "\n".join(["<image>"] * len(views))
    + "\nDescribe the scene and reason about the spatial relationships "
      "between the visible objects."
)
generation_config = {
    "do_sample": False,
    "max_new_tokens": 512,
}

response = model.chat(
    processor.tokenizer,
    pixel_values,
    question,
    generation_config,
    num_patches_list=[1] * len(views),
    vggt_images=vggt_images,
)
print(response)

</details>

For the architecture, training procedure, and evaluation protocol, see the paper.

Type: internvl_vggt_chat
License: mit
Created: Aug 2, 2026
Updated: Aug 14, 2026