Zamba2-VL-2.7B

Zamba2-VL-2.7B

Zamba2-VL is a family of vision-language models built on Zyphra's Zamba2 LLM suite. It supports single- and multi-image understanding and grounding, achieving state-of-the-art performance among multimodal models of similar size while benefiting from the inference efficiency of the Zamba2 architecture. You can find all models in the Zamba2-VL family here.

Zamba2-VL uses the Mistral v0.1 tokenizer and was trained on 100B tokens of vision-text and pure text data sourced from open web-datasets.

Zamba2-VL-2.7B is based on Zamba2-2.7B LLM and uses the Qwen2.5-VL vision encoder as vision backbone.

Performance

Zamba2-VL-2.7B performs strongly against models of comparable size and inference FLOPs, outperforming several larger models as well. Its small compute and memory footprint make it an ideal generalist model for on-device applications.

All numbers are run on the Zyphra evaluation harness (based on VLMEvalKit). Other models are ordered by total parameter count. Bold indicates the best score in each row, while <u>underlined</u> values indicate the lowest score.

Quick start

Prerequisites

To use Zamba2-VL, install zamba2-vl branch from our fork of transformers library, which is based on the v4.57.1 of transformers:

pip install "transformers @ git+https://github.com/Zyphra/transformers.git@zamba2-vl"
pip install qwen-vl-utils==0.0.2
pip install flash_attn

The command above relies on requirements for transformers v4.57.1 being installed in your environment. If you're installing in a fresh Python environment, you might want to specify a specific extra, like [dev-torch], to install all the dependencies:

pip install "transformers[dev-torch] @ git+https://github.com/Zyphra/transformers.git@zamba2-vl"

For the fastest setup, ensure your environment matches an existing flash_attn wheel, otherwise the installation will build from source.

To install dependencies necessary to run Mamba2 kernels, install mamba-ssm from source (due to compatibility issues with PyTorch) as well as causal-conv1d:

pip install --no-build-isolation "causal-conv1d @ git+https://github.com/Zyphra/z-causal-conv1d.git@zamba2-vl"
pip install --no-build-isolation "mamba-ssm @ git+https://github.com/Zyphra/mamba.git@zamba2-vl"

You can run the model without using the optimized Mamba2 kernels, but it is not recommended as it will result in significantly higher latency and memory usage.

Inference

from transformers import Zamba2_VLForConditionalGeneration, Zamba2_VLProcessor
import torch
from PIL import Image
from qwen_vl_utils import process_vision_info
import requests

device = "cuda"
processor = Zamba2_VLProcessor.from_pretrained("Zyphra/Zamba2-VL-2.7B", temporal_patch_size=1)
model = Zamba2_VLForConditionalGeneration.from_pretrained("Zyphra/Zamba2-VL-2.7B", device_map=device, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
question = "What do you see in the image? Give us some detail."
num_img_tokens = 3400

conversation = [
    {"role": "user", "content": [
        {"type": "image", "image": image, "max_pixels" : num_img_tokens * 28 * 28, "min_pixels" : 10 * 28 * 28},
        {"type": "text", "text": question},
      ]
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
images, _ = process_vision_info(conversation)
inputs = processor(text=prompt, images=images, add_special_tokens=True, return_tensors="pt")
inputs = {key: value.to(device) for key, value in inputs.items()}

outputs = model.generate(**inputs, max_new_tokens=100)
print(processor.tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))