CASA-Helium1-VL-2B

<img align="right" src="readme_images/CASA.png" width="150px" >

Model Card for CASA-Helium1-VL-2B

CASA (Project Page . arXiv . github) stands for Cross-Attention over Self-Attention. CASA is a VLM based entirely on cross-attention (CA) with enjoys practical benefits for streaming tasks by never adding image tokens to the KVCache of the text model.

Specifically, CASA layers inject visual tokens into a text stream by using image-to-text cross-attention, which act in smaller local attention windows. Simply put, at inference, the model only looks at the latest seen image explicitly. We show that this setup can be naturally leveraged streaming inference e.g. on video inputs, enabling almost constant memory time and inference cost For qualitative samples of CASA used for live video captioning, please check the associated HuggingFace space.

Model Details

Model Description

This model page contains the model weights for CASA trained from a pretrained text-only Helium1-2B backbone and from the image encoder from Qwen2.5-VL-3B. In the collection, we also provides weights for:

• CASA-Qwen2_5-VL-3B: A CASA model adapted from the full pretrained Qwen2.5-VL-3B (keeping the backbone LLM weights are kept frozen)
• CASA-Qwen2_5-VL-3B-LiveCC: A CASA model adapted from the full pretrained Qwen2.5-VL-3B and futher finetuned for live video captioning.
• Helium1-VL-2B: A reference VLM trained from Helium1-2B with standard token insertion mechanism in the same setting as CASA-Helium1-VL-2B.

Model Summary:

• Developed by: Kyutai
• Model type: Multimodal vision+text model based on Cross-Attention
• Language(s) (NLP): English
• License: CC-BY-NC-SA-4.0
• LLM Backboner from: Helium1 2B
• Image Encoder from: Qwen2.5-VL 3B
• Terms of use: As the released models include frozen weights of the Qwen2.5VL-3B image encoder, the weights are subject to the Qwen RESEARCH LICENSE AGREEMENT

Model Sources

• Project Page kyutai.org/casa
• Preprint arXiv
• Repository: Github kyutai-labs/casa

Uses

Direct Use

The intended use of the Helium model is research and development of vision-language systems, including but not limited to image or video understanding.

CASA-Helium1-VL-2B, Helium1-VL-2B and CASA-Qwen2_5-VL-2B can be used as vision-language models to analyze or interpret images as input signals.

CASA-Qwen2_5-VL-2B-LiveCC can be used as a vision-language model on streaming videos as inputs at 2fps.

The models can be used primarly with English as a language. For most downstream use cases, the model should be aligned with supervised fine-tuning, RLHF or related methods.

Out-of-Scope Use

The model should not be used in other languages than the ones on which it was trained. The model is not intended to be used to impersonate other people or any malicious use of any kind.

Bias, Risks, and Limitations

Our CASA-Helium1 model was not aligned to human preferences. As such, the model can generate incorrect, biased, harmful or generally unhelpful content. Thus, the model should not be used for downstream applications without further alignment, evaluations and mitigations of risks.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

See our github repository for additional scripts to perform benchmark evaluation and live video captioning.

Below is a short snippet to show you how to load our models, process inputs, and run inference, using a standard HuggingFace transformers pipeline and chat template.

# Minimal requirements:
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "rich",
#     "einops>=0.8.1",
#     "torch==2.7.0",
#     "transformers==4.51.3",
#     "torchvision==0.22.0",
#     "flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.0.post2/flash_attn-2.8.0.post2+cu12torch2.7cxx11abiTRUE-cp310-cp310-linux_x86_64.whl"
# ]
# ///
import torch
from transformers.models.auto.modeling_auto import AutoModel
from transformers.models.auto.processing_auto import AutoProcessor    

model_id = "kyutai/CASA-Helium1-VL-2B"
model = AutoModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).cuda()
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "assets/casa_model.png",
            },
            {
                "type": "text",
                "text": "Describe this image.",
            },
        ],
    },
]
inputs = processor.tokenize_messages(messages=conversation)
inputs = inputs.to(model.device)
input_len = inputs["input_ids"].shape[1]
output_ids = model.generate_from_image(
  **inputs, 
  max_new_tokens=512,
  pre_image_tokens=processor.pre_image_tokens,
  post_image_tokens=processor.post_image_tokens,
  eos_token_id=model.generation_config.eos_token_id,
)[0, input_len:]
response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)

Training Details

Please have a look at our associated research paper for details on the training pipeline.

Training Data

To train our CASA-Helium models we use the FineVision dataset as well as a small, non overlapping, subset of Llava-OneVision-1.5-Instruct

Evaluation

We evaluate our models on a range of benchmarks covering document understanding (DocVQA), chart understanding (ChartQA, InfoVQA), visual text reading (TextVQA, OCRBench), and general QA (RealWorldQA, AI2D, GQA, MME). Results are reported below. Please refer to our project page and arxiv paper for additional evaluation.

<table style="border-collapse: collapse;"> <tr> <th rowspan="2" align="left">Model</th> <th colspan="3" align="center">Document / Chart</th> <th colspan="2" align="center">Scene Text</th> <th colspan="4" align="center">Knowledge / QA</th> </tr> <tr> <th>ChartQA</th> <th>DocVQA</th> <th>InfoVQA</th> <th>OCRBench</th> <th>TextVQA</th> <th>RealWorldQA</th> <th>AI2D</th> <th>GQA</th> <th>MME</th> </tr> </thead> <tbody> <tr> <td align="left">Helium1-VL-2B</td> <td>81.6</td><td>89.1</td><td>61.8</td> <td>728</td><td>75.5</td> <td>59.9</td><td>67.7</td><td>55.5</td><td>1732</td> </tr> <tr> <td align="left"><span style="color:#fb923c;"><strong>CASA-Helium1-VL-2B</strong></span></td> <td>73.4</td><td>83.7</td><td>48.6</td> <td>723</td><td>71.0</td> <td>58.3</td><td>63.3</td><td>54.6</td><td>1572</td> </tr> <tr> <td align="left"><span style="color:#60a5fa;">mPLUG-Owl3 8B</span></td> <td>59.2^†</td><td>55.9^†</td><td>36.8^†</td> <td>527^†</td><td>69.0</td> <td>63.9^†</td><td>73.4</td><td>65.0</td><td>1940^†</td> </tr> <tr> <td align="left"><span style="color:#60a5fa;">mPLUG-Owl3 2B</span></td> <td>48.5^†</td><td>48.2^†</td><td>28.1^†</td> <td>450^†</td><td>62.6</td> <td>56.9^†</td><td>62.6</td><td>61.0</td><td>1551^†</td> </tr> </tbody> </table> <p> ^† Reproduced with the publicly available models on Hugging Face. &ensp; <!-- ◇ Results and model not publicly available. --> </p> <p align="center"> <em> Results for <code>CASA-Helium1-VL-2B</code> compared to a recent cross-attention baseline (blue), and our token insertion (<code>Helium1-VL-2B</code> trained in the same conditions. CASA outperforms current SoTA cross-attention-based VLMs, narrowing the gap to insertion-based approaches. </em> </p> <table style="border-collapse: collapse;"> <thead> <tr> <th rowspan="2" align="left">Model</th> <th colspan="3" align="center">Document / Chart</th> <th colspan="2" align="center">Scene Text</th> <th colspan="4" align="center">Knowledge / QA</th> </tr> <tr> <th>ChartQA</th> <th>DocVQA</th> <th>InfoVQA</th> <th>OCRBench</th> <th>TextVQA</th> <th>RealWorldQA</th> <th>AI2D</th> <th>GQA</th> <th>MME</th> </tr> </thead> <tbody> <tr> <td align="left"> Qwen2.5-VL-3B </td> <td>84.0</td><td>93.6</td><td>77.1</td> <td>797</td><td>79.3</td> <td>62.2^†</td><td>81.6</td><td>61.0^†</td><td>2249^†</td> </tr> <tr> <td align="left"> <span style="color:#fb923c;"><strong>CASA-Qwen2_5-VL-3B</strong></span> </td> <td>82.4</td><td>88.9</td><td>59.6</td> <td>790</td><td>77.4</td> <td>62.5</td><td>75.1</td><td>59.4</td><td>1918</td> </tr> </tbody> </table> <p> ^† Reproduced with the publicly available models on Hugging Face. </p> <p align="center"> <em> Results for <code>CASA-Qwen2_5-VL-3B</code>, adapted from frozen Qwen2.5-VL. CASA reaches performance close to the original insertion-based model while while training only the CASA layers and last blocks of the image encoder. </em> </p>

Technical Specifications

Compute Infrastructure

CASA-Helium1-2B was trained starting from a Helium1-2B LLM and the image encoder from Qwen2.5-VL-3B. We finetune the whole LLM backbone as well as the last four blocks of the image encoder. The currently released model was trained on four DGX nodes with 8 H100 GPUs.

Software

Our training code and inference code was implemented in Pytorch.

Citation

@article{kyutai2025casa,
  author = {Moritz Böhle and Amélie Royer and Juliette Marrie and Edouard Grave and Patrick Pérez},
  year = {2025},
  title = {CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion},
  journal = {ArXiv},
  url = {https://arxiv.org/abs/2512.19535}
}

Model Card Authors and Contact

• Amelie Royer
• Moritz Boehle
• Juliette Marrie