vroid-single-view-ablation

vroid-single-view-ablation (12k iter)

Single-ego ablation checkpoint for the multiview Cosmos-Predict2.5 2B model, fine-tuned on the dhyun22/ego_gen_vroid_single SYNTHETIC egocentric dataset (VRoid avatars, 2-person interactions rendered in synthetic scenes).

This is the synthetic-data counterpart of kimhosoo/comind-single-view-ablation (same ablation, trained on the real CoMind dataset instead) — same architecture, same training recipe, same base checkpoint, only the data source differs. Meant for direct real-vs-synthetic comparison.

What this ablation is

The base Cosmos-Predict2.5 multiview model generates two ego views jointly — leader and helper attend to each other's tokens (self-attention) and captions (cross-attention) in one forward pass, conditioned on reference frames, cross-view Plücker rays, and shared warp/pose context.

This checkpoint instead generates each ego view completely independently (V=1):

• No reference frames — appearance/pose/camera conditioning from in-context reference frames removed.
• Own-view-only warp — re-sourced from the wearer's own past frames only (no cross-view pooling); VRoid's warp is fully re-computed (warp_mode=own_only_recomputed), not merely masked.
• Self-only pose — only the wearer's own skeleton is rendered into the pose condition.
• Per-view canonical camera — Plücker rays canonicalized to each view's own frame-0 (w2c[0] = I).
• V=1 pooling — the two ego views (..._ego_a / ..._ego_b) are pooled as independent training samples rather than concatenated into one V=2 forward pass. Self-attention structurally cannot see the other view's tokens; cross-attention only ever sees that one view's own caption.

Training

• Base checkpoint: nvidia/Cosmos-Predict2.5-2B (auto/multiview/524af350-...-_ema_bf16.pt), strict_resume=False.
• Data: dhyun22/ego_gen_vroid_single, ~8,168 train / 1,144 val independent V=1 samples (avatar-identity-disjoint split, manifest/{train,val}_avatar_disjoint.csv). Caption: 70% the dataset's real per-clip detail_all.csv.bak action caption, 30% fallback to the fixed generic caption (stochastic mix, mirrors this codebase's existing caption_probability convention).
• Text encoder: nvidia/Cosmos-Reason1-7B (Qwen2.5-VL-7B architecture, NVIDIA reasoning fine-tune).
• 2x NVIDIA H200, FSDP shard size 2, 12,000 iterations (single continuous run, no resume needed).
• Loss at iter 12,000: ~0.058 (rectified-flow loss, noisy per-iteration; comparable to the CoMind run's ~0.057 at the same iteration count).
• Checkpoints saved every 500 iterations; this upload is the final iter_000012000 checkpoint.

Files

• model_ema_bf16.pt — EMA weights in bfloat16 (recommended for inference; converted from the training-time DCP/FSDP checkpoint via scripts/convert_distcp_to_pt.py in the cosmos-predict2.5 repo).
• code_changes.tar.gz — same archive as kimhosoo/comind-single-view-ablation (identical code powers both the real and synthetic runs). Extract into cosmos_predict2/_src/ of a clone of nvidia-cosmos/cosmos-predict2.5. Includes:
  • predict2_multiview/datasets/comind_pairs.py — ComindSingleEgoDataset + its single_view V=1 pooling mode.
  • predict2_multiview/datasets/vroid_pairs.py — VroidSingleEgoDataset, used by this checkpoint's data loader. Reads the already-per-view single.npz layout directly, mixes core/detail captions stochastically.
  • predict2_multiview/configs/vid2vid/experiment/nymeria_pose_2actor.py — adds the comind_single_ego, comind_single_view, and vroid_single_view (this checkpoint's experiment) configs; fixes an unset job.group (fails Config.validate()), a private local text-encoder checkpoint path (now resolved via nvidia/Cosmos-Reason1-7B on the Hub), and trainer.validation_iter (was left at its effectively-disabled default).
  • predict2_multiview/callbacks/nymeria_validation_viz.py — adds comind_single_ego and vroid_single_ego branches so the periodic validation-preview-video callback can build these dataset types.
  • predict2_multiview/configs/vid2vid/config.py — registers the vroid_pairs.py dataloaders.

Usage

git clone https://github.com/nvidia-cosmos/cosmos-predict2.5.git
cd cosmos-predict2.5
tar -xzf code_changes.tar.gz --strip-components=2 -C cosmos_predict2/_src

torchrun --nproc_per_node=1 -m scripts.train \
  --config=cosmos_predict2/_src/predict2_multiview/configs/vid2vid/config.py \
  -- experiment=vroid_single_view \
  checkpoint.load_path=/path/to/model_ema_bf16.pt \
  trainer.max_iter=0  # inference / sampling only, no further training

See the experiment docstring (vroid_single_view in nymeria_pose_2actor.py) for the full conditioning contract and the dhyun22/ego_gen_vroid_single dataset README for the .single.npz field layout.