video2vla-robotwin15-ground-truth-latent-slice-idm

Video2VLA RoboTwin-15 Ground-Truth Latent-Slice IDM

A compact inverse-dynamics action head trained on frozen ground-truth HunyuanVideo 1.5 causal-VAE latents from 15 RoboTwin tasks. This is an action model, not a video generator.

Model

• Input: BF16/FP32 latent tensor [B, 32, 6, H, W] (training cache used H=19, W=38).
• Temporal layout: two context latent slices followed by four future latent slices.
• Output: normalized joint targets [B, 24, 14].
• Architecture: the same 4.408M-parameter spatial decoder is applied independently to every latent slice; each slice predicts four actions.
• No proprioception input.

The training cache was created before Hunyuan posttraining and contains clean VAE encodings. This run did not use generated inference latents, a posttrained LoRA, text encoders, or online VAE encoding. Training added independent flow-style noise sampled uniformly from [0.0, 0.2]; validation used fixed noise 0.1.

Training

• Dataset: 15-task RoboTwin randomized-500 wide corpus.
• Usable indexed windows: 69,704 train / 8,821 validation.
• Steps: 100,000.
• Effective batch: 4 on one NVIDIA GH200.
• Optimizer: AdamW, LR 3e-4, weight decay 0.05, 500 warmup steps.
• Runtime: BF16; fused optimizer.
• Source commit: d76f4a79a8a7f38f3e280dcc9d112d7c3592f25b.
• Weights & Biases run

Validation

Metrics are computed on normalized joint targets. They should not be interpreted as task success rates.

Files

• checkpoint_100000.pt: model, optimizer, scheduler, resolved config, and step; suitable for resuming with Video2VLA.
• checkpoint_030000.pt: earlier 30k checkpoint retained for comparison.
• model.safetensors: model state only.
• config.yaml: source training configuration.
• resolved_config.json: fully resolved configuration embedded in the checkpoint.
• action_stats.json: joint-action normalization statistics.
• training_metadata.json: provenance, metrics, and SHA-256 hashes.
• training_code/: relevant source snapshot used by this cache-backed run.

Loading the model-only weights

from safetensors.torch import load_file
from video2vla.action_heads import LatentSliceHead

model = LatentSliceHead(
    latent_channels=32,
    action_dim=14,
    hidden_channels=128,
    spatial_pool=8,
    mlp_dim=512,
    actions_per_latent=4,
    dropout=0.05,
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

The outputs are normalized joint positions; use action_stats.json to invert z-score normalization.

Additional mixed-latent checkpoint

The repository also contains the newer 60% clean / 40% generated-latent IDM, trained for 100,000 steps. It is stored separately so checkpoint_100000.pt at the repository root continues to identify the original clean-only model.