openvla-oft-libero-gguf

OpenVLA-OFT - LIBERO (GGUF for vla.cpp)

GGUF conversion of moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 for inference with vla.cpp, a lightweight C++ inference engine for Vision-Language-Action models built on top of llama.cpp.

OpenVLA-OFT (Optimized Fine-Tuning) is a 7B-scale VLA: a Prismatic-style VLM with a fused DINOv2-L/14-reg4 + SigLIP-SO400m/14 vision backbone (224 px) over a Llama-2-7B language backbone, coupled to an MLPResNet (L1 regression) action head that predicts an 8-step action chunk in parallel (no autoregressive decoding). It consumes two camera views (third-person + wrist) and an 8-D proprioceptive state. This checkpoint is a single multi-task fine-tune covering all four LIBERO suites - spatial, object, goal, and 10 (LIBERO-Long). The vision tower is baked into the combined GGUF, so no separate mmproj file is needed.

Files

Usage

Build vla-server from the vla.cpp repo:

git clone https://github.com/VinRobotics/vla.cpp && cd vla.cpp
bash ./patches/patch.sh                        # fetch + patch llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release      # use a CUDA build for GPU inference
cmake --build build -j"$(nproc)"

Then serve and drive an episode. Select the LIBERO suite via VLA_OPENVLA_OFT_UNNORM_KEY (this picks the action-unnormalisation stats on the server, and must match the suite you evaluate):

# Terminal 1 - serve. No mmproj argument (vision is baked into the combined GGUF).
VLA_OPENVLA_OFT_UNNORM_KEY=libero_object_no_noops \
    ./build/vla-server --bind tcp://*:5557 \
    openvla-oft-libero.gguf

# Terminal 2 - drive a LIBERO episode (inside the LIBERO uv venv)
VLA_OPENVLA_OFT_UNNORM_KEY=libero_object_no_noops \
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
    --arch openvla_oft \
    --task libero_object --task-id 0 --n-episodes 10 \
    --n-action-steps 8 \
    --stats-json dataset_statistics.json \
    --vla-addr tcp://localhost:5557

Notes:

• openvla_oft runs at 224 px with two views, an 8-step action chunk (--n-action-steps 8), and proprio state dim 8.
• Set VLA_OPENVLA_OFT_UNNORM_KEY on both server and client to the same suite (libero_spatial_no_noops, libero_object_no_noops, libero_goal_no_noops, or libero_10_no_noops). The server uses it for action un-normalisation and the client for proprio normalisation; their built-in defaults differ, so always set it explicitly. For the other suites pass the matching --task (libero_spatial, libero_goal, libero_10).
• Pass --stats-json dataset_statistics.json (proprio normalisation). The same stats are also baked into the GGUF.
• The tokenizer auto-loads from the base checkpoint; pass --tokenizer /path/to/ckpt to load it offline.

Benchmark

Smoke test - libero_object task 0, 10 episodes, vla-server + run_sim_client_direct.py:

The upstream OpenVLA-OFT paper reports ≈97–98% averaged across the four LIBERO suites; the full vla.cpp sweep (per suite, 10 tasks × 20 episodes) will be filled in once measured on reference hardware.

License

Weights follow the upstream license of moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 (MIT, inherited from OpenVLA). The vla.cpp conversion tooling and inference engine are MIT-licensed.