Downloads · 30 days
0
VR-VLA/VR-egoverse-v06-pretrained-moe-lr1e4-bs64-60k-snapshot
VR-egoverse-v06-pretrained-moe-lr1e4-bs64-60k-snapshot is a robotics model from VR-VLA. Use it for the robotics task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as other.
Egocentric pretraining checkpoint for a vision-language-action (VLA) model: a frozen Qwen3.5-0.8B backbone with LoRA adapters, plus a cross-attention flow-matching action head, trained on narrated egocentric human vid…
Downloads · 30 days
0
Access
Public
Updated Oct 9, 2026
Repo size
14.1 GB
Likes
0
Public
Click a slice to open those files.
.pt14.1 GB · 100%
From the Hugging Face model README
Egocentric pretraining checkpoint for a vision-language-action (VLA) model: a frozen Qwen3.5-0.8B backbone with LoRA adapters, plus a cross-attention flow-matching action head, trained on narrated egocentric human video. This variant adds MoE action head.
This is a pretraining checkpoint, not a deployable policy. Its action space is 153-dim two-handed human motion. It cannot drive a robot directly and requires a finetune onto the target embodiment's action space.
It is also a mid-run snapshot. The run is a 100,000-step cosine schedule, paused at step 87,500. The latest file is
ckpt_87.5k.pt(step 87,500, learning rate 3.81e-06); the earlierckpt_60k.ptis kept for comparison. At step 60,000 where the learning rate is still 3.46e-05 — about 35% of peak. It is published for comparison against the 60k baseline below, not because training had converged; the final-decay phase is missing. The run continues past this point.
The action head's dense FFN is replaced in every one of the 24 blocks by a
sparse mixture of experts: 16 routed SwiGLU experts with top-4 routing plus 1 shared
expert. Routing follows DeepSeek-V3 — per-expert sigmoid affinities, selection on
affinity + bias, mixing on the raw affinity — with loss-free load balancing (the bias is a
buffer updated by rule after each optimizer step, never by gradient) and a router z-loss of
1e-3. Load stayed balanced throughout: no dead experts, max/mean expert load ~1.50.
EgoVerse v06 — 35,000 narrated egocentric clips, ~1.16M training windows, 16-step action chunks at 10 Hz. Narration is text supervision generated by a larger vision-language model.
153 dimensions per timestep:
[ head_d_pos(3) | head_d_rot 6d(6)
| left hand: pos_cam(3) rot_cam 6d(6) kp21(63)
| right hand: pos_cam(3) rot_cam 6d(6) kp21(63) ]
The head block is anchor-relative; hand blocks are per-step camera-frame wrist poses plus 21 wrist-relative keypoints. Rotations use the Zhou et al. 6d representation.
⚠️ action_norm_egoverse.json is included and required. All targets are in normalized
space; the checkpoint is meaningless without these statistics.
| base model | Qwen/Qwen3.5-0.8B (frozen) + LoRA r=16 |
| action head | 24 cross-attention blocks, hidden 1024, flow matching (10-step Euler) |
| timestep conditioning | adaRMS (adarms) |
| action head FFN | sparse MoE, 16 routed experts (top-4) + 1 shared, router z-loss 1e-3 |
| steps | ckpt_87.5k.pt: 87,500; ckpt_60k.pt: 60,000 (of a 100,000-step schedule) |
| learning rate | 1e-4, cosine decay over 100,000 steps, 100-step warmup |
| effective batch | 64 (16 × accum 4), single GPU |
| precision | bf16 autocast, fp32 master weights |
| checkpoint | 2162 tensors, 586M parameters |
Trained with two objectives: a language objective over the narration and a flow-matching objective over the action chunks.
Mean over the 2,500 steps ending at 87,500 (ckpt_87.5k.pt; training-set losses, no held-out evaluation of the pretraining objective). The losses at 60,000 are in this repo's history.
| action (flow matching) | 0.0477 |
| narration (LM) | 0.0984 |
| total (action + 0.1×narration [+ observation]) | 0.0575 |
| router z-loss (added outside the total) | 0.0082 |
VR-egoverse-v06-pretrained-lr1e4-bs64-60k is the checkpoint every finetune in this project starts from. It is not a matched control for this run: it used 60,000 steps with its cosine ending there, warmup 1500, additive timestep conditioning and a 4-step sampler, whereas this run uses the current recipe (100,000 steps, warmup 100, adaRMS, 10-step sampler). A matched plain control on this recipe (100k, adaRMS, atomic captions) has since finished. Loss values are therefore not comparable across the two.
Initialization for downstream robot policy finetuning, where the action-space projections are replaced for the target embodiment and the learned representation is reused.
Not intended for direct deployment, and not evaluated as a standalone policy.
Derived from EgoVerse (internal; see dataset terms) and Qwen3.5-0.8B. Users are responsible for complying with both. This checkpoint grants no rights beyond them.