Downloads · 30 days
0
luuuulinnnn/egohoi-control-15k-2026-09-03
egohoi-control-15k-2026-09-03 is a machine learning model from luuuulinnnn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as other.
The two EgoHOI control checkpoints continued from 13000 to 15000 steps, on a frozen Wan 2.1 I2V-14B-720P base, plus the full inference sweep and PSNR/SSIM evaluation for both. Run date 2026-09-03.
Downloads · 30 days
0
Access
Public
Updated Sep 4, 2026
Repo size
36.6 GB
Likes
0
Public
Click a slice to open those files.
.pt35.1 GB · 96%
From the Hugging Face model README
The two EgoHOI control checkpoints continued from 13000 to 15000 steps, on a frozen Wan 2.1 I2V-14B-720P base, plus the full inference sweep and PSNR/SSIM evaluation for both. Run date 2026-09-03.
Code: edisiondyli/egohoi_extension, branch obj_rope, commit 6ea62eb (recorded as
repo=6ea62eb branch=obj_rope in each run's slurm-*.out).
Both runs resume the step-13000 checkpoints published in
luuuulinnnn/egohoi-control-hot3d-h2o,
which were trained at commit 563354c. train.steps is absolute and global_step is restored from
the checkpoint, so each loop ran exactly steps 13001 → 15000 — 2000 new steps per run, nothing
retrained from scratch.
| run | camera | hand | obj | view_render_root | final.pt |
|---|---|---|---|---|---|
train_camera_obj | ✅ | ❌ | ✅ | hand_view_render | 12.91 GiB |
train_camera_egohoihand_obj | ✅ | ✅ | ✅ | view_render_old | 19.77 GiB |
Caveat, inherited from the 13k runs. These two runs differ in two variables at once — the camera render set (
hand_view_rendervsview_render_old) and whether the egohoi hand encoder is enabled. Neither run's advantage over the other can be attributed to one of those alone. This repo continues both runs unchanged; it does not fix that confound.
Generated at 81 frames, 40 sampling steps, seed 23, bf16, then compared to the ground-truth clip over
the same frame interval at 480×480. HOT3D is the held-out hot3d_preprocessed_test grid (96 clips ×
start frames 0 and 69); H2O is subject3_ego.
| run | PSNR (dB) | SSIM | n | HOT3D PSNR / SSIM (n) | H2O PSNR / SSIM (n) |
|---|---|---|---|---|---|
train_camera_obj | 20.7565 | 0.66204 | 235 | 21.0121 / 0.66636 (190) | 19.6772 / 0.64380 (45) |
train_camera_egohoihand_obj | 19.2436 | 0.65170 | 238 | 19.6631 / 0.66048 (190) | 17.5830 / 0.61694 (48) |
Aggregates (with standard deviations) are in metrics/psnr_ssim_summary.json; per-video rows are in
metrics/<infer_run>_per_video.csv.
The two n differ, and not only by size.
train_camera_objhas 45 H2O videos,train_camera_egohoihand_objhas 48 —subject3_ego__o2__{5,6,7}are missing fromtrain_camera_obj. HOT3D is 190 in both, out of a possible 192:clip-002973andclip-003354produced a single interval each, identically in both runs. So even the two 15k rows above are not scored over an identical video set; the HOT3D columns are, the H2O columns are not.
No frozen-Wan baseline here. Wan 2.1 inference was not run for this suite. There is no
infer_wan21_baseline/directory. The baseline in the 13k repo (12.230 dB / 0.3686) was measured on a different video set and is not carried over.
| 13k (old set) | 15k (new set) | |
|---|---|---|
camera_obj | 20.4601 / 0.65707, n=139 | 20.7565 / 0.66204, n=235 |
camera_egohoihand_obj | 18.7084 / 0.63700, n=142 | 19.2436 / 0.65170, n=238 |
The 13k numbers were scored on the 139 / 142-video set from the earlier suite; these 15k numbers
are scored on the 235 / 238-video set. Different clips, different intervals (the 15k sweep
generates two HOT3D start frames per clip, 0-80 and 69-149), different H2O coverage. Do not read
the difference between those two columns as a training effect. It is dominated by which videos are
in each average.
Restricting both scorings to the H2O videos present at 13k and 15k under the same clip + interval
key gives a strictly paired comparison — same clips, same intervals, same ground truth, only the
checkpoint differs:
| run | metric | 13k | 15k | Δ | t | improved |
|---|---|---|---|---|---|---|
camera_obj | PSNR (dB) | 19.7937 | 19.6772 | −0.1165 | −2.54 | 17 / 45 |
camera_obj | SSIM | 0.65093 | 0.64380 | −0.00712 | −4.92 | 9 / 45 |
camera_egohoihand_obj | PSNR (dB) | 17.4001 | 17.5830 | +0.1830 | +2.03 | 31 / 48 |
camera_egohoihand_obj | SSIM | 0.61137 | 0.61694 | +0.00557 | +2.47 | 32 / 48 |
So the honest reading of 2000 extra steps: camera_obj got very slightly worse, camera_egohoihand_obj
got very slightly better, and both moves are within a fifth of a dB. The t-statistics are large only
because pairing removes almost all the between-clip variance, not because the effect is large.
The learning-rate schedule explains it. Resuming with train.steps=15000 rebuilds the cosine schedule
over 15000 total steps, so at step 13001 the camera parameter group restarts at ≈4.3e-7 and
decays to 0 by step 15000 — about 4% of the 1e-5 base at the start of the leg. The first logged
step in train.log reads lr=4.318e-07 and the last reads lr=0.000e+00, in both runs.
This is not a misconfiguration. It is exactly the learning rate a run natively configured for 15000
steps would be using over its final 2000 steps; this leg simply finishes that curve. It does mean the
camera branch — the one pixel metrics are most sensitive to — moved very little here, while the obj
LoRA (and, in the second run, the hand encoder and Wan LoRA) kept training at a constant 1e-5. The
measured deltas are the size that setup predicts, and no large movement should be read into them.
Per-group LR curves are in each run's tensorboard/ (train/lr_camera, train/lr_obj, plus the hand
and Wan-LoRA groups in the second run).
train_camera_obj/
checkpoints/final.pt # step 15000, full training checkpoint (12.91 GiB)
resolved_config.yaml # the fully resolved config actually used
train.log # per-step loss / lr / grad_norm for steps 13001-15000
slurm-487547.out # scheduler log; carries "repo=6ea62eb branch=obj_rope"
tensorboard/ # 2000 train points + 4 validation points, 9 scalar series
train_camera_egohoihand_obj/
checkpoints/final.pt # step 15000 (19.77 GiB)
resolved_config.yaml, train.log, slurm-487550.out, tensorboard/
infer_train_camera_obj/ # 235 mp4 + 141 inference_summary.json
hot3d/<clip>/<clip>_<span>_camera_obj_inference.mp4
h2o/<subject>__<action>__<idx>/<same-stem>_0-80_camera_obj_inference.mp4
<clip-dir>/inference_summary.json # checkpoint, encoders, caption, GT paths, seed, precision
infer_train_camera_egohoihand_obj/ # 238 mp4 + 144 inference_summary.json
same shape, *_camera_hand_obj_inference.mp4
metrics/
psnr_ssim_summary.json # overall + per-dataset aggregates
infer_train_camera_obj_per_video.csv # 235 rows
infer_train_camera_egohoihand_obj_per_video.csv # 238 rows
eval.log # the eval job's log
Compare both training curves at once:
tensorboard --logdir .
final.pt is a full training checkpoint — model + Adam state + scheduler + scaler + RNG — not
merged weights, and it is the exact file every inference run loaded (each inference_summary.json
records its path). Roughly two thirds of each file is Adam optimizer state — one exp_avg and one exp_avg_sq tensor
per trainable weight — so the trainable weights themselves account for about a third of each size.
Weights are stored fp32 even though training ran bf16 mixed precision.
checkpoints/step_00015000.pt — content-identical to final.pt (every zip member's CRC32 and
size matches; only the archive root name differs). A pure duplicate.checkpoints/step_00014000.pt — an intermediate resume point, never evaluated.slurm-*_{0..7}.out — redundant with the inference_summary.json files.wandb/ — offline run state, redundant with tensorboard/, and containing symlinks that point
outside the tree.psnr_db_globalThe per-video CSVs have header
dataset,clip,interval,psnr_db,psnr_db_global,ssim,frames. The headline numbers in this card and in
psnr_ssim_summary.json come from psnr_db (per-frame PSNR averaged over the clip).
psnr_db_global is a second, differently-pooled PSNR that appears only in the CSVs — do not mix the
two.
Paths below are the ones actually used; PY is the repo venv interpreter and the working directory is
the repo root.
PY=/scratch/project/prj-02-phai-lab/lulin/jichen/egohoi_extension/.venv/bin/python
cd /scratch/project/prj-02-phai-lab/lulin/jichen/egohoi_extension
module load CUDA/12.6.0 # CUDA_HOME must point at a real toolkit
RESUME is the 13k checkpoint from the earlier suite (outputs_v2/<run>/checkpoints/final.pt); on
a requeue the launcher instead picks the newest local step_*.pt.
train_camera_obj:
OUT=.../outputs_v4/train_camera_obj
RESUME=.../outputs_v2/train_camera_obj/checkpoints/final.pt
$PY scripts/train.py --resume "$RESUME" \
--set camera_encoder.enable=true \
--set obj_encoder.enable=true \
--set hand_encoder.enable=false \
--set obj_encoder.rope="tss-rope" \
--set _allow_legacy_obj_lora_config=true \
--set obj_encoder.target_modules='["self_attn.q","self_attn.k","self_attn.v"]' \
--set obj_encoder.train_attn_blocks=null \
--set obj_encoder.injected_layers=null \
--set hot3d_data.view_render_root="hand_view_render" \
--set h2o_data.view_render_root="hand_view_render" \
--set h2o_data.raw_root="outputs/h2o_preprocessed" \
--set hot3d_data.num_workers=2 \
--set train.steps=15000 \
--set logging.loaded_data.enabled=false \
--set logging.tensorboard.enabled=true \
--set logging.tensorboard.log_dir="$OUT/tensorboard" \
--set logging.wandb.name=train_camera_obj \
--set output_dir="$OUT" \
--requested-gpus 8
train_camera_egohoihand_obj — identical except three flags:
--set hand_encoder.enable=true \
--set hot3d_data.view_render_root="view_render_old" \
--set h2o_data.view_render_root="view_render_old" \
The four legacy obj-LoRA flags are mandatory. The 13k checkpoints were trained at commit 563354c,
where the object LoRA was built as self_attn.{q,k,v}_image_lora across all 40 blocks. At HEAD
6ea62eb the default geometry changed — k,v on blocks 20–39 only — so a strict load of the 13k
weights fails. These four restore the checkpoint's geometry:
--set _allow_legacy_obj_lora_config=true
--set obj_encoder.target_modules='["self_attn.q","self_attn.k","self_attn.v"]'
--set obj_encoder.train_attn_blocks=null
--set obj_encoder.injected_layers=null
Also note train.steps is absolute, not "steps to add": 15000 with a restored global_step of
13000 yields 2000 steps, and it is what rebuilds the cosine schedule discussed above.
Run as an 8-way array per checkpoint: tasks 0–6 shard the 96 HOT3D test clips (passed as repeated
--hot3d-clip flags from shards_v4/hot3d_<N>.txt), task 7 does H2O. infer_batch.py skips items
already recorded in a clip's summary, so a requeue resumes.
$PY scripts/infer_batch.py \
--checkpoint .../outputs_v4/train_camera_obj/checkpoints/final.pt \
--datasets "$DS" $CLIPS \
--hot3d-root outputs/hot3d_preprocessed_test \
--hot3d-view-render-root "hand_view_render" \
--h2o-view-render-root "hand_view_render" \
--h2o-raw-root outputs/h2o_preprocessed \
--output-dir .../outputs_v4/infer_train_camera_obj
For train_camera_egohoihand_obj, swap the checkpoint, the output dir, and both view-render roots to
view_render_old.
--hot3d-root outputs/hot3d_preprocessed_test is required. It defaults to
outputs/hot3d_preprocessed, which is the training root; omitting it silently evaluates on training
clips.
One CPU-only job over both inference trees (PSNR/SSIM is cv2 + skimage):
$PY eval_psnr_ssim.py \
.../outputs_v4/infer_train_camera_obj \
.../outputs_v4/infer_train_camera_egohoihand_obj \
--out .../outputs_v4/metrics
This writes psnr_ssim_summary.json and one <infer_run>_per_video.csv per tree. Note that
scripts/eval.py in the upstream repo is empty (0 bytes); the working per-pair tool is
scripts/compute_video_psnr_ssim.py, which eval_psnr_ssim.py drives over a whole run.
h2o_data.raw_root — H2O intrinsics and poses ship inside
outputs/h2o_preprocessed/<subject>/<seq>/cam4/, not at the configured data/h2o; pass
--set h2o_data.raw_root=outputs/h2o_preprocessed (training) and --h2o-raw-root (inference).object_render_metadata.json stores absolute /home/jc/... paths. Only
frames[].image_path is read by the training and inference loaders, so that field must be rewritten
to outputs/h2o_preprocessed/... or every H2O object window is silently dropped.CUDA_HOME must point at a real toolkit, or transformers → deepspeed raises
MissingCUDAException at startup.Identical across both runs except hand_encoder.enable and the two view_render_root values — the
resolved configs differ in exactly five places, those three plus output_dir and the logging names.
| Base model | Wan 2.1 I2V-14B-720P (frozen), 40 attention blocks |
| Steps | 15000 absolute — this leg ran 13001–15000 |
| Optimizer | AdamW, lr 1e-5, weight_decay 0 |
| LR this leg | camera group cosine ≈4.3e-7 → 0; obj / hand / wan_lora constant 1e-5 |
| Precision | bf16 + gradient checkpointing |
| Datasets | HOT3D + H2O interleaved — 19 HOT3D steps : 1 H2O step |
| Batch size | 1 per GPU, 8 GPUs |
| Video | 81 frames @ 480×480 |
| HOT3D start frames | [0, 10, 20, 30, 40, 50, 60, 69] |
| Validation | every 500 steps, fixed timesteps [357, 682, 833, 921, 978], noise seed 23 |
| Checkpointing | every 1000 steps |
| obj rope | tss-rope |
| Object stages | A 1500 / B 1500 / C remainder |
| Seed | 23 |
| Hardware | 8× NVIDIA H200, 16 CPUs per node |
| module | configuration |
|---|---|
camera_encoder | ControlNet-style residuals into Wan blocks 0–19; Plücker rays (256 hidden, 6 ch); control_dim 1024, 20 layers, 16 heads, ffn 4096, conv_out 5120, rms_norm_across_heads |
obj_encoder | Stand-In image-branch LoRA r128 / α128, zero init, on self_attn.{q,k,v} across all 40 blocks (legacy geometry) |
hand_encoder | Wan LoRA r128 / α128 on q,k,v,o,ffn.0,ffn.2 (kaiming); Conv3d hand stack (concat_dim 4) + reference hand-frame injection — enabled only in train_camera_egohoihand_obj |
Recorded in every inference_summary.json: sampling_steps=40, seed=23, precision=bf16,
frame_count=81. HOT3D intervals 0-80 and 69-149; H2O 0-80.
Preprocessed HOT3D + H2O from EndeavourDD/Jichen.
outputs/hot3d_preprocessed + H2O subject1_ego, subject2_egooutputs/hot3d_preprocessed_test (96 clips, the config's val_clips grid, packaged
separately) + H2O subject3_egooutputs/hot3d_qwen_captions.json,
outputs/h2o_preprocessed/h2o_action_label_captions.json| Base model | Wan 2.1 I2V-14B-720P (frozen) |
| Datasets | HOT3D + H2O |
| Code | edisiondyli/egohoi_extension, branch obj_rope, commit 6ea62eb |
| Continues from | luuuulinnnn/egohoi-control-hot3d-h2o — the step-13000 checkpoints, trained at commit 563354c |
| Related | luuuulinnnn/egohoi-layer-ablation-hot3d-h2o — a separate 15000-step layer-injection suite |
| Hardware | 8× NVIDIA H200 per training job (nodes dgx071, dgx075); single-GPU shards for inference |
| Training runtime | train_camera_obj 07:28:23 · train_camera_egohoihand_obj 08:16:30 |
| Inference runtime | 8 single-GPU shards each — 02:58:10 and 03:37:39 |
| Evaluation runtime | 00:28:25, CPU only |
| Wan 2.1 baseline | not run for this suite |
Files here embed absolute cluster paths (/scratch/project/prj-02-phai-lab/lulin/jichen/...) in
train.log, the slurm logs, the resolved configs, and every inference_summary.json. The
Resumed training state from .../outputs_v2/<run>/checkpoints/final.pt line in each train.log is the
primary on-disk evidence of the 13k → 15k continuation, and is left intact.