egohoi-control-hot3d

luuuulinnnn/egohoi-control-hot3d

egohoi-control-hot3d is a machine learning model from luuuulinnnn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as other.

Conditional egocentric world-model training on HOT3D, built on a frozen Wan 2.1 I2V-14B-720P DiT. Seven runs covering all combinations of the three trainable conditioning modules (cameraencoder, handencoder, objencode…

Downloads · 30 days

Access

Public

Updated Jul 27, 2026

Repo size

85.2 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pt84.9 GB · 100%

At a glance

Library: pytorch
License: other
Access: Public
Created: Jul 27, 2026
Updated: Jul 27, 2026
SHA: 2ce5310d

Model card

From the Hugging Face model README

EgoHOI Control — HOT3D (Wan 2.1 I2V-14B)

Conditional egocentric world-model training on HOT3D, built on a frozen Wan 2.1 I2V-14B-720P DiT. Seven runs covering all combinations of the three trainable conditioning modules (camera_encoder, hand_encoder, obj_encoder), trained with scripts/train.py from the obj_rope branch of edisiondyli/egohoi_extension.

All 7 runs completed the full 8000 steps.

Results

Sorted by validation loss (lower is better). Validation uses fixed timesteps + deterministic noise, so values are directly comparable across runs.

run	camera	hand	obj	GPUs	val loss	train cum-avg	steps
`train_all_encoders`	✅	✅	✅	8×H200	0.05562	0.12364	8000
`train_camera_hand`	✅	✅	❌	4×H200	0.05840	0.12395	8000
`train_camera_obj`	✅	❌	✅	4×H200	0.06153	0.12524	8000
`train_camera_only`	✅	❌	❌	4×H200	0.06196	0.12604	8000
`train_hand_obj`	❌	✅	✅	4×H200	0.07644	0.12815	8000
`train_hand_only`	❌	✅	❌	4×H200	0.07668	0.12837	8000
`train_obj_only`	❌	❌	✅	4×H200	0.08045	0.13067	8000

Two clear trends: more conditioning modules → lower validation loss, and the camera encoder contributes the most (every camera-enabled run beats every camera-disabled run).

Repository layout

<run_name>/
  checkpoints/          # model checkpoints
  train.log             # full training log (loss, lr, grad_norm, validation)
  slurm-<jobid>.out     # scheduler stdout
  resolved_config.yaml  # the exact resolved config used for the run
  tensorboard/          # training curves (events.out.tfevents.*)
  wandb/                # offline W&B run

View the curves for all runs at once:

tensorboard --logdir .

Training setup

Identical across all 7 runs except the three enable flags.


Base model	Wan 2.1 I2V-14B-720P (frozen)
Steps	8000
Optimizer	AdamW, lr 1e-5, weight_decay 0
Precision	bf16 + gradient checkpointing
Batch size	1 per GPU
Video	81 frames @ 480×480, start_frames [0, 34, 69]
Validation	every 400 steps, 50 clips, timesteps [357, 682, 833, 921, 978], deterministic noise
Checkpointing	every 400 steps
Seed	23
Hardware	NVIDIA H200 (2 CPUs per GPU), ~25–33 h per run

Conditioning modules

module	configuration
`camera_encoder`	ControlNet-style residuals into the first 20 Wan blocks; Plücker rays (256 hidden, 6 ch) + masks; control_dim 1024, 20 layers, 16 heads, ffn 4096, rms_norm
`hand_encoder`	Wan LoRA r128 / α128 on `q,k,v,o,ffn.0,ffn.2` (kaiming init); Conv3d hand stack (concat_dim 4) + reference hand-frame injection
`obj_encoder`	Stand-In image-branch LoRA r128 / α128 on `self_attn.{q,k,v}` (zero init), all attention blocks; staged curriculum 1500 + 1500

Data

Preprocessed HOT3D clips (EndeavourDD/Jichen), 1516 clips total.

Train: clips ≥ 1949 (568 distinct clips observed, range 1949–3364)
Validation: 50 clips starting at 1849 (range ~1849–1898)
Disjoint — clips 1899–1948 are held out as a gap between the two splits.

Captions come from hot3d_qwen_captions.json (4548 windows keyed as clip-XXXXXX_sNNN_eNNN).

Notes

Checkpoints are full training checkpoints (model + optimizer state), not merged weights.
final.pt is the end of training (step 8000). best.pt tracks the lowest single-step training loss and best_average.pt the lowest running average — note that single-step diffusion loss is noisy, so final.pt is generally the checkpoint to use.

Library: pytorch
License: other
Created: Jul 27, 2026
Updated: Jul 27, 2026