Downloads · 30 days
46
7% of all-time downloads
Eyeline-Labs/Vista4D
Vista4D is a video-to-video model from Eyeline-Labs. Use it for the video-to-video task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
[](https://eyeline-labs.github.io/Vista4D)
Downloads · 30 days
46
7% of all-time downloads
All-time downloads
630
Public
Repo size
42.1 GB
Likes
33
Public
Click a slice to open those files.
.pth42.1 GB · 100%
From the Hugging Face model README
Kuan Heng Lin<sup>1,3∗</sup>, Zhizheng Liu<sup>1,4∗</sup>, Pablo Salamanca<sup>1,2</sup>, Yash Kant<sup>1,2</sup>, Ryan Burgert<sup>1,2,5∗</sup>, Yuancheng Xu<sup>1,2</sup>, Koichi Namekata<sup>1,2,6∗</sup>, Yiwei Zhao<sup>2</sup>, Bolei Zhou<sup>4</sup>, Micah Goldblum<sup>3</sup>, Paul Debevec<sup>1,2</sup>, Ning Yu<sup>1,2</sup> <br/> <sup>1</sup>Eyeline Labs, <sup>2</sup>Netflix, <sup>3</sup>Columbia University, <sup>4</sup>UCLA, <sup>5</sup>Stony Brook University, <sup>6</sup>University of Oxford<br>
<sup>∗</sup>Work done during an internship at Eyeline Labs
<div align="center"> <video controls autoplay muted style="width: 100%;" src="https://media.githubusercontent.com/media/Eyeline-Labs/Vista4D/website/media/vista4d.mp4"></video> </div>Vista4D is a video reshooting framework which synthesizes the dynamic scene represented by an input source video from novel camera trajectories and viewpoints. We bridge the distribution shift between training and inference for point-cloud-grounded video reshooting, as Vista4D is robust to point cloud artifacts from imprecise 4D reconstruction of real-world videos by training on noisy, reconstructed multiview videos. Our 4D point cloud with temporally-persistent static points also explicitly preserves scene content and improved camera control. Vista4D generalizes to real-world applications such as dynamic scene expansion (casual video capture of scene as background reference), 4D scene recomposition (point cloud editing), and long video inference with memory.
This is the Hugging Face repository containing our model weights. We provide two Vista4D checkpoints finetuned on Wan-AI/Wan2.1-T2V-14B(https://huggingface.co/Wan-AI/Wan2.1-T2V-14B):
| Checkpoint | Base model | Training resolution | Training steps | Notes |
|---|---|---|---|---|
384p49_step=30000 | Wan2.1-T2V-14B | 672 × 384, 49 frames | 30000 | N/A |
720p49_step=3000 | Wan2.1-T2V-14B | 1280 × 720, 49 frames | 3000 | Finetuned from 384p49_step=30000 |
To do Vista4D inference, first download the Wan 2.1 and Vista4D checkpoints to ./checkpoints/. The Vista4D checkpoints are hosted on Eyeline-Labs/Vista4D. Download both the 384p and 720p checkpoints into ./checkpoints/vista4d/ with
hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d
If you only need one resolution, pass --include to grab just that variant with
hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d --include "384p49_step=30000/*" OR "720p49_step=3000/*"
You'll also need the Wan2.1-T2V-14B base model. Download it from Wan-AI/Wan2.1-T2V-14B into ./checkpoints/wan/Wan2.1-T2V-14B/ with
hf download Wan-AI/Wan2.1-T2V-14B --local-dir ./checkpoints/wan/Wan2.1-T2V-14B
Instructions on how to use these weights, more results, and paper can be found on our project page and GitHub repository.