VideoChat3-4B

VideoChat3-4B

VideoChat3-4B is the 4B-parameter release of VideoChat3, a fully open, efficient, and generalist Video Multimodal Large Language Model for video understanding. The model is trained with a scalable video data synthesis and curation pipeline covering general, long-form, and streaming video scenarios. VideoChat3 achieves strong performance across mainstream video understanding benchmarks and surpasses prior open-source models at comparable or larger parameter scales, while releasing model weights, training code, training strategy, and training datasets to support reproducible open research.

📄 Paper · 🌐 Homepage · 💻 GitHub · 🤗 Paper Page

<p align="center"> <img src="https://raw.githubusercontent.com/MCG-NJU/VideoChat3/main/docs/public/paper/overview.png" alt="VideoChat3 Overview" width="100%"> </p>

Installation

pip install torch transformers accelerate qwen-vl-utils
pip install decord opencv-python-headless

For faster inference, FlashAttention 2 can be used if it is correctly installed in your environment:

pip install flash-attn --no-build-isolation

Quick Start

The following examples show how to run image and video inputs. For a complete runnable script, please refer to demo_vc3.py.

Image/Video Inference

from transformers import AutoModelForCausalLM, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "MCG-NJU/VideoChat3-4B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

# image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/image.jpg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
# video
# messages = [
#     {
#         "role": "user",
#         "content": [
#             {"type": "video", "video": "/path/to/video.mp4"},
#             {"type": "text", "text": "What happens in this video?"},
#         ],
#     }
# ]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

images, videos, video_kwargs = process_vision_info(
    messages,
    image_patch_size=14,
    return_video_kwargs=True,
    return_video_metadata=True,
)
inputs = processor(
    text=text,
    images=images,
    videos=videos,
    do_resize=False,
    return_tensors="pt",
    **(video_kwargs or {}),
)
inputs = inputs.to(model.device)
if hasattr(model, "dtype"):
    inputs = inputs.to(model.dtype)
generated_ids = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
)
generated_ids = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.tokenizer.batch_decode(
    generated_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

Online Proactive Response

VideoChat3-4B supports proactive response for streaming video understanding. The proactive response protocol uses three explicit states:

• </Silence>: the current frame is irrelevant to the user question, or no meaningful event has started yet.
• </Standby>: a relevant event is unfolding, but the model needs more future frames before giving a reliable answer. When </Standby> is predicted, the next frame is processed with a larger visual budget.
• </Response>: the model has collected enough visual evidence and provides the answer.

Example usage with the provided proactive demo:

python demo_vc3_proactive.py \
  --model MCG-NJU/VideoChat3-4B \
  --video /path/to/video.mp4 \
  --question "your question"

Citation

@article{li2026videochat3,
  title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding},
  author={Li, Xinhao and Zhu, Yuhan and Zeng, Xiangyu and Dong, Yuhao and Wu, Haoning and Zhang, Zhiqiu and Yang, Yuandong and Ma, Changlian and Zhang, Qingyu and Shi, Yansong and others},
  journal={arXiv preprint arXiv:2607.14935},
  year={2026}
}