I3D-ViT

I3D-ViT

Introduction

I3D-ViT is the standalone vision encoder obtained after the vision encoder pre-training stage (Stage 0) described in the paper. It contains the VideoChat3VisionModel implementation, vision configuration, and the ViT weights. This vision encoder is subsequently used to initialize the visual backbone of MCG-NJU/VideoChat3-4B.

📄 Paper · 🌐 Homepage · 💻 GitHub · 🤗 Paper Page

Example usage

For image input:

from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel

model_path = "MCG-NJU/I3D-ViT"
image_path = "example.jpg"

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
).eval()
processor = AutoImageProcessor.from_pretrained(model_path)

image = Image.open(image_path).convert("RGB")
inputs = processor(images=image, return_tensors="pt")

pixel_values = inputs["pixel_values"].to(
    device=model.device,
    dtype=model.dtype,
)
grid_thws = inputs["image_grid_thw"].to(model.device)

with torch.inference_mode():
    image_features: list = model(
        pixel_values=pixel_values,
        grid_thws=grid_thws,
    )

print(image_features[0].shape)

For video input:

import torch
from transformers import AutoModel, AutoVideoProcessor


model_path = "MCG-NJU/I3D-ViT"
video_path = "example.mp4"
num_frames = 4

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
).eval()
processor = AutoVideoProcessor.from_pretrained(
    model_path,
    trust_remote_code=True,
)

inputs = processor(
    videos=video_path,
    fps=None,
    num_frames=num_frames,
    return_tensors="pt",
)

pixel_values = inputs["pixel_values_videos"].to(
    device=model.device,
    dtype=model.dtype,
)
grid_thws = inputs["video_grid_thw"].to(model.device)

with torch.inference_mode():
    video_features: list = model(
        pixel_values=pixel_values,
        grid_thws=grid_thws,
    )

print(video_features[0].shape)

Citation

@article{li2026videochat3,
  title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding},
  author={Li, Xinhao and Zhu, Yuhan and Zeng, Xiangyu and Dong, Yuhao and Wu, Haoning and Zhang, Zhiqiu and Yang, Yuandong and Ma, Changlian and Zhang, Qingyu and Shi, Yansong and others},
  journal={arXiv preprint arXiv:2607.14935},
  year={2026}
}