CASA-Qwen2_5-VL-3B-LiveCC

CASA-Qwen2_5-VL-3B-LiveCC

This repository contains model weights for CASA-Qwen2_5-VL-3B-LiveCC, a Qwen-2.5VL model adapted from token insertion to a cross-attention-based architecture using CASA layers. It has been further finetuned on the LiveCC dataset for live video captioning.

• Paper: CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
• Project Page: kyutai.org/casa
• Repository: github.com/kyutai-labs/casa

Please refer to the main model card for more information and instructions to run. We provide model weights for other CASA models in the associated model collection.

Quick Start

Below is a short snippet to show you how to load the model, process inputs, and run inference using the 🤗 transformers library. Note that this model requires trust_remote_code=True.

import torch
from transformers.models.auto.modeling_auto import AutoModel
from transformers.models.auto.processing_auto import AutoProcessor

model_id = "kyutai/CASA-Qwen2_5-VL-3B-LiveCC"
model = AutoModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).cuda()
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/your/image.png",
            },
            {
                "type": "text",
                "text": "Describe this image.",
            },
        ],
    },
]
inputs = processor.tokenize_messages(messages=conversation)
inputs = inputs.to(model.device)
input_len = inputs["input_ids"].shape[1]
output_ids = model.generate_from_image(
  **inputs,
  max_new_tokens=512,
  pre_image_tokens=processor.pre_image_tokens,
  post_image_tokens=processor.post_image_tokens,
  eos_token_id=model.generation_config.eos_token_id,
)[0, input_len:]
response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)

Live video Captioning

In our [github repository], we provide a script to caption a video using our CASA-Qwen2_5-VL-3B-LiveCC model and generate the resulting video with subtitles embedded at the actual time they are generated.

Note that you will also need to install ffmpeg for this script to run. The Python dependencies are handled with uv

# Script options
uv run scripts/gen_livecc_subtitles.py --help
# Generation with Qwen2.5VL+CASA
uv run scripts/gen_livecc_subtitles.py --sample_path path_to_video.mp4 --srt True --temp 0.0

Additional qualitative samples are available in the associated HuggingFace space.

<div align="center"> <p align="center" width="100%"> <video src="https://github.com/user-attachments/assets/cb205fe2-11fb-4e8d-98ac-e1a250e5573b" width="80%" controls></video> </p> <p> The input video is taken from the Animal Kingdom dataset, and the subtitles displayed are generated with <code>CASA-Qwen2_5-VL-3B-LiveCC</code>.

Specifically, video frames are extracted at 2fps, and subtitles are displayed in real-time at the timestamp they are generated< </p>

<p><small> <i><b>Transcript:</b> "This video shows a fox in the Arctic. The Arctic is an area of Earth that's covered by ice and snow year -round, and it gets very cold there. Foxes are adapted to live in this cold environment because they have a thick layer of fur to keep them warm when they're out in the snow. This fox is walking through the snow and looking around for food or maybe just for safety from predators like wolves or bears that might be around. Foxes are also known for their ability to jump really high and"</i></small></p> </div

Citation

If you use CASA in your research, please cite our work:

@article{kyutai2025casa,
  author = {Moritz B\"ohle and Am\'elie Royer and Juliette Marrie and Edouard Grave and Patrick P\'erez},
  year = {2025},
  title = {CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion},
  journal = {ArXiv},
  url = {https://arxiv.org/abs/2512.19535}
}