vla-jepa-libero

VLA-JEPA - LIBERO (GGUF for vla.cpp)

GGUF conversion of lerobot/VLA-JEPA-LIBERO for inference with vla.cpp, a lightweight C++ inference engine for Vision-Language-Action models built on top of llama.cpp.

VLA-JEPA is a 3B VLA built on a Qwen3-VL-2B-Instruct vision-language backbone

• a 24-layer ViT (1024d, patch 16, temporal-patch 2, spatial-merge ÷2, deepstack features fused from layers 5/11/17, 256 px) feeding a 28-layer Qwen3 LM (2048d, 16 query / 8 KV heads × 128, RoPE θ=5e6) - coupled to a DiT-B flow-matching action head (16 transformer blocks, 12 heads × 64, inner 768d, cross-attending the 2048d LM stream, 1024d output) that denoises a 7-step action chunk over 4 flow-matching steps from 32 learned future tokens (action_dim 7, state_dim 8).

The upstream checkpoint also carries a V-JEPA world-model predictor, but it is not on the action-inference path - so the conversion drops the world model and keeps only the Qwen3-VL backbone + DiT action head. The vision tower is baked into the combined GGUF, so no separate mmproj file is needed.

Files

Required normalisation stats (not bundled here)

VLA-JEPA un-normalises with LeRobot processor stats, not a dataset_statistics.json. The client reads two safetensors from the dir passed to --stats-json:

Both ship in the upstream lerobot/VLA-JEPA-LIBERO repo. Copy them next to vla-jepa.gguf (then --stats-json . works) or point --stats-json at a local copy of the upstream checkpoint.

Usage

Build vla-server from the vla.cpp repo, then:

# Terminal 1 - serve (use the CUDA build for inference). No mmproj argument.
VLA_JEPA_BF16_WEIGHTS=1 ./build-cuda/vla-server --bind tcp://*:5566 \
    vla-jepa.gguf

# Terminal 2 - drive a LIBERO episode (inside the LIBERO uv venv)
python eval/client/run_sim_client_direct.py \
    --arch vla_jepa \
    --task libero_object --task-id 0 --n-episodes 10 \
    --n-action-steps 7 \
    --stats-json . \
    --vla-addr tcp://localhost:5566

Notes:

• vla_jepa runs at 256 px with a 7-step action chunk (--n-action-steps 7) and proprio state dim 8.
• Set VLA_JEPA_BF16_WEIGHTS=1 to keep BF16 matmuls (default upcasts weights to F32). VLA_NUM_STEPS=<n> overrides the 4 flow-matching steps.
• --stats-json <dir> is required - point it at the dir holding the two policy_{pre,post}processor safetensors (see above).
• The tokenizer/processor auto-loads from Qwen/Qwen3-VL-2B-Instruct on the Hub (the client expands it with the 28 <|action_i|> + <|embodied_action|> tokens). Pass --tokenizer /path/to/Qwen3-VL-2B-Instruct to load it offline.

Benchmark

Smoke test - LIBERO sweep, vla-server + run_sim_client_direct.py:

VRAM and latency have not been measured yet; they will be filled in once profiled on reference hardware.

License

The upstream lerobot/VLA-JEPA-LIBERO repo declares no explicit license. Review the upstream terms and the licenses of its components - Qwen3-VL-2B (Apache-2.0) and V-JEPA (Meta research license) - before use. The vla.cpp conversion tooling and inference engine are Apache-2.0-licensed.