egohoi-control-hot3d-h2o

EgoHOI Control — HOT3D + H2O (Wan 2.1 I2V-14B)

Conditional egocentric world-model training on HOT3D + H2O (interleaved 19:1), built on a frozen Wan 2.1 I2V-14B-720P DiT. Code: edisiondyli/egohoi_extension @ branch obj_rope (commit 563354c).

Both training runs completed the full 13000 steps. Both were then evaluated against a frozen-Wan baseline on the same validation grid.

Results

Validation loss (training objective)

Validation uses fixed timesteps [357, 682, 833, 921, 978] with deterministic noise, so the two numbers are directly comparable.

Generation quality — PSNR / SSIM vs ground truth

All videos generated at 40 sampling steps, unipc, guide_scale 5.0, seed 23, then compared to the GT clip over the same frame interval at 480×480.

Both conditioned models beat the frozen baseline by a wide margin (+6.5 to +8.2 dB PSNR, +0.27 to +0.29 SSIM), confirming the camera/hand/object conditioning is doing real work rather than the base model carrying the result.

Caveat on exp1 vs exp2. These two runs differ in two variables at once — the camera render set (hand_view_render vs view_render_old) and whether the egohoi hand encoder is enabled. exp1's advantage therefore cannot be attributed to the render set alone; a single-variable ablation would be needed to separate the two effects.

Baseline caveat. The frozen Wan baseline is image-to-video from the first GT frame with a text prompt only — it receives no camera, hand, or object conditioning, and nothing anchors it to the GT trajectory. Its low PSNR/SSIM reflects that missing conditioning, not a broken model. It was run at Wan's 480p setting (832*480) to match the fine-tuned runs' resolution.

Repository layout

<run_name>/                         # train_camera_obj, train_camera_egohoihand_obj
  checkpoints/                      # final.pt (step 13000), best.pt, best_average.pt
  train.log, slurm-*.out, resolved_config.yaml, submit.sbatch
  tensorboard/                      # training curves
  wandb/                            # offline W&B run

infer_<run>/                        # infer_camera_obj, infer_camera_egohoihand_obj,
  hot3d/<clip>/<clip>_<span>.mp4    # infer_wan21_baseline
  h2o/<clip>/<clip>_<span>.mp4      # all generated videos
  inference_summaries.jsonl         # GT <-> generated pairing, caption, seed, checkpoint
  slurm-*.out, submit.sbatch

metrics/
  psnr_ssim_summary.json            # aggregate, overall + per dataset
  <run>_per_video.csv               # per-video PSNR/SSIM
  eval_psnr_ssim.py                 # the driver used to produce them

View curves for both training runs at once:

tensorboard --logdir .

Training setup

Identical across both runs except the three enable flags and view_render_root.

Conditioning modules

Data

Preprocessed HOT3D (1516 clips) + H2O (4 subjects) from EndeavourDD/Jichen (outputs/).

• HOT3D train: clips ≥ 1849 excluding the validation grid
• HOT3D val: 100 explicit clip ids sampled with random.seed(23) over [1849, 3364]
• H2O train: subject1_ego, subject2_ego · H2O val: subject3_ego
• Captions: hot3d_qwen_captions.json, h2o_action_label_captions.json

Reproduction notes

Fixes required to run this outside the author's machine:

1. h2o_data.raw_root — the H2O camera intrinsics/poses (cam_intrinsics.txt, cam_pose/) ship inside outputs/h2o_preprocessed/<subject>/<seq>/cam4/, not at the configured data/h2o. Set --set h2o_data.raw_root=outputs/h2o_preprocessed.
2. H2O object_render_metadata.json stores absolute image_path values (/home/jc/jc/outputs/...), unlike HOT3D which uses repo-relative paths. They must be rewritten to outputs/h2o_preprocessed/... or every H2O object window is silently dropped.
3. scripts/eval.py is empty (0 bytes). The working metric tool is scripts/compute_video_psnr_ssim.py, which compares one pair at a time; metrics/eval_psnr_ssim.py here drives it over a whole run.
4. scripts/wan21_inference.py hardcodes CUDA_VISIBLE_DEVICES=7, /home/jc/jc/... paths, and 720p. It also writes no summary JSON, so the metric driver falls back to deriving GT pairs from the output filenames.
5. CUDA_HOME must point at a real CUDA toolkit (module load CUDA/12.6.0), otherwise transformers → deepspeed raises MissingCUDAException on the first generated item.

Notes

• Checkpoints are full training checkpoints (model + optimizer state), not merged weights. final.pt (step 13000) is the one to use; best.pt / best_average.pt track noisy single-step and running-average training loss.
• Both training runs exceeded the 48 h scheduler limit and were resumed from step checkpoints; the slurm-*.out files cover the separate legs.
• exp1 generated 139 unique videos rather than 143: the val_clips list contains 4 duplicate HOT3D ids (2024, 2604, 2780, 3022). The baseline is 142/145 — one clip (clip-001877) failed to generate on every attempt.