JointControlvideo

JointControlVideo — MANO Hand-Conditioned Video Generation

Official checkpoint for "Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints" (ECCV 2026).

Given a starting frame and a sequence of 3D MANO hand-joint trajectories, this checkpoint generates an egocentric video that follows the prescribed hand motion — conditioned via occlusion-aware, 3D geometric hand-joint embeddings injected directly into the latent space of Wan2.1-I2V-14B-480P, instead of dense 2D tracks or a separate pose tokenizer.

Files

Both are trained on top of a frozen Wan2.1-I2V-14B-480P backbone; you still need the base model's DiT/VAE/text-encoder/CLIP weights, which are pulled automatically from Wan-AI/Wan2.1-I2V-14B-480P the first time you run the pipeline.

Usage

Set up the code from ZhangCYG/JointControlVideo

Citation

@inproceedings{zhang2026controllable,
  title     = {Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints},
  author    = {Zhang, Chenyangguang and Ye, Botao and Chen, Boqi and Delitzas, Alexandros and Wang, Fangjinhua and Pollefeys, Marc and Wang, Xi},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

License

Apache 2.0.