cosmos3-nano-droid-fd-wm

Cosmos3-Nano DROID Forward-Dynamics World Model

Action-conditioned forward-dynamics (FD) world model for the DROID manipulation setup, post-trained from Cosmos3-Nano. Given a single conditioning frame plus a chunk of robot actions, it predicts the resulting multi-view video — usable as a rollout environment for policy evaluation / dreaming.

• Base model: Cosmos3-Nano (Qwen3-VL-8B MoT backbone + Wan2.2 VAE tokenizer)
• Mode: forward_dynamics — condition on frame 0 + all actions → predict video
• Action space: joint_pos (8D absolute joint targets), use_state on
• Views: multi-view concat (wrist + 2 exterior cameras)
• Chunk length: 32 · operating point: resolution 480, shift 5
• Precision: bf16

Checkpoints

Each subfolder is a self-contained HF safetensors export (7 shards + config.json + checkpoint.json).

Held-out eval = seeded 1% split (never trained on), n=12 episodes, motion-PSNR masked to moving regions. The single-chunk metric plateaus ~5k iters; the extra training in latest does not improve it and does not add long-horizon/off-manifold robustness — best is the recommended default; latest is provided for completeness.

Training

• g512: 16× H200 (2 nodes), global batch 512, DROID (92,233 episodes, multi-view).
• Post-trained from the base Cosmos3-Nano DCP checkpoint via the cosmos-framework action SFT recipe.

Usage

These are cosmos-framework model exports (not transformers-native). Load with the cosmos-framework inference stack (the exported config.json / checkpoint.json describe the OmniMoTModel). Point the DROID multiview FD rollout tooling at a subfolder as the checkpoint path.

Caveats

Research checkpoints. FD conditions on a single history frame, so autoregressive rollouts drift over long horizons and can collapse off-manifold (many re-conditioning steps) — coherent window ≈ 30–60 frames at keep-10 cadence, longer at chunk-32 cadence. Best for short-horizon / qualitative policy checks.