Downloads · 30 days
0
VD-IT
buxiangzhiren/VD-IT
VD-IT is a machine learning model from buxiangzhiren. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as ecl-2.0.
Access
Public
Updated Oct 23, 2024
Repo size
10.5 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pth10.5 GB · 100%
At a glance
- License
- ecl-2.0
- Access
- Public
- Created
- Sep 28, 2024
- Updated
- Oct 23, 2024
- SHA
- f60c8d14
Model card
From the Hugging Face model README
VD-IT model
The is our pre-trained checkpoint for our paper Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation.
We use a video diffusion model (ModelScopeT2V) as our base model, applying prompt tuning to adapt it as a visual backbone for downstream video understanding tasks.
Model traning
We first pre-train our model on Ref-COCO and then fine-tune it on Ref-YouTube-VOS. The training of the models utilizes two NVIDIA A100 GPUs, processing 5 frames per clip over the course of 9 epochs. The initial learning rate is set to 5e-5 and reduced by a factor of 10 at the 6th and 8th epochs.
- License
- ecl-2.0
- Created
- Sep 28, 2024
- Updated
- Oct 23, 2024