Downloads · 30 days
0
VR-VLA/VR-egoverse-v06-pretrained-rtc-lr1e4-bs64-60k-snapshot
VR-egoverse-v06-pretrained-rtc-lr1e4-bs64-60k-snapshot is a robotics model from VR-VLA. Use it for the robotics task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as other.
Egocentric pretraining checkpoint for a vision-language-action (VLA) model: a frozen Qwen3.5-0.8B backbone with LoRA adapters, plus a cross-attention flow-matching action head, trained on narrated egocentric human vid…
Downloads · 30 days
0
Access
Public
Updated Oct 9, 2026
Repo size
13 GB
Likes
0
Public
Click a slice to open those files.
.pt13 GB · 100%
From the Hugging Face model README
Egocentric pretraining checkpoint for a vision-language-action (VLA) model: a frozen Qwen3.5-0.8B backbone with LoRA adapters, plus a cross-attention flow-matching action head, trained on narrated egocentric human video. This variant adds real-time chunking.
This is a pretraining checkpoint, not a deployable policy. Its action space is 153-dim two-handed human motion. It cannot drive a robot directly and requires a finetune onto the target embodiment's action space.
It is also a mid-run snapshot. The run is a 100,000-step cosine schedule, paused at step 77,500. The latest file is
ckpt_77.5k.pt(step 77,500, learning rate 1.20e-05); the earlierckpt_60k.ptis kept for comparison. At step 60,000 where the learning rate is still 3.46e-05 — about 35% of peak. It is published for comparison against the 60k baseline below, not because training had converged; the final-decay phase is missing. The run continues past this point.
Trained with real-time chunking (RTC): an observation predictor shifts the
vision hidden states from observation time to execution time, conditioned on the committed
actions (motion features through a temporal attention block, then zero-initialised FiLM) and
supervised against the backbone's hidden states for the frame d steps ahead. A random delay
d in [0, 15] is drawn per window, the first d chunk steps enter the head as clean actions
(prefix pinning) and the flow loss covers only the postfix. With the switch off the model is
byte-for-byte the plain architecture.
EgoVerse v06 — 35,000 narrated egocentric clips, ~1.16M training windows, 16-step action chunks at 10 Hz. Narration is text supervision generated by a larger vision-language model.
153 dimensions per timestep:
[ head_d_pos(3) | head_d_rot 6d(6)
| left hand: pos_cam(3) rot_cam 6d(6) kp21(63)
| right hand: pos_cam(3) rot_cam 6d(6) kp21(63) ]
The head block is anchor-relative; hand blocks are per-step camera-frame wrist poses plus 21 wrist-relative keypoints. Rotations use the Zhou et al. 6d representation.
⚠️ action_norm_egoverse.json is included and required. All targets are in normalized
space; the checkpoint is meaningless without these statistics.
| base model | Qwen/Qwen3.5-0.8B (frozen) + LoRA r=16 |
| action head | 24 cross-attention blocks, hidden 1024, flow matching (10-step Euler) |
| timestep conditioning | adaRMS (adarms) |
| async execution | on — observation predictor, prefix pinning, proprio roll-forward |
| steps | ckpt_77.5k.pt: 77,500; ckpt_60k.pt: 60,000 (of a 100,000-step schedule) |
| learning rate | 1e-4, cosine decay over 100,000 steps, 100-step warmup |
| effective batch | 64 (16 × accum 4), single GPU |
| precision | bf16 autocast, fp32 master weights |
| checkpoint | 962 tensors, 542M parameters |
Trained with two objectives: a language objective over the narration and a flow-matching objective over the action chunks, plus an MSE against the execution-time vision hidden states that trains the observation predictor only.
Mean over the 2,500 steps ending at 77,500 (ckpt_77.5k.pt; training-set losses, no held-out evaluation of the pretraining objective). The losses at 60,000 are in this repo's history.
| action (flow matching) | 0.0350 |
| narration (LM) | 0.1150 |
| observation predictor | 0.6607 |
| total (action + 0.1×narration [+ observation]) | 0.7072 |
VR-egoverse-v06-pretrained-lr1e4-bs64-60k is the checkpoint every finetune in this project starts from. It is not a matched control for this run: it used 60,000 steps with its cosine ending there, warmup 1500, additive timestep conditioning and a 4-step sampler, whereas this run uses the current recipe (100,000 steps, warmup 100, adaRMS, 10-step sampler). A matched plain control on this recipe (100k, adaRMS, atomic captions) has since finished. Loss values are therefore not comparable across the two.
Initialization for downstream robot policy finetuning, where the action-space projections are replaced for the target embodiment and the learned representation is reused.
Not intended for direct deployment, and not evaluated as a standalone policy.
Derived from EgoVerse (internal; see dataset terms) and Qwen3.5-0.8B. Users are responsible for complying with both. This checkpoint grants no rights beyond them.