Ivy-VL-llava

<code>Ivy-VL</code> is a lightweight multimodal model with only 3B parameters. 

It accepts both image and text inputs to generate text outputs. 

Thanks to its lightweight design, it can be deployed on edge devices such as AI glasses and smartphones, offering low memory usage and high speed while maintaining strong performance on multimodal tasks. Some well-known small models include PaliGemma 3B, Moondream2, Qwen2-VL-2B, InternVL2-2B, and InternVL2_5-2B. Ivy-VL outperforms them on multiple benchmarks.

Model Summary:

• Developed: AI Safeguard, CMU, Standford

• Model type: Multi-modal model (image+text)

• Language: Engligh and Chinese

• License: Apache 2.0

• Architecture: Based on LLaVA-One-Vision

• LLM: Qwen/Qwen2.5-3B-Instruct

• Vision Encoder: google/siglip-so400m-patch14-384

• Notebook demo: Ivy-VL-demo.ipynb

Evaluation：

Most of the performance data comes from the VLMEvalKit leaderboard or the original papers. We conducted evaluations using VLMEvalKit. Due to differences in environments and the LLMs used for evaluation, there may be slight variations in performance.

How to use:

# pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
from llava.model.builder import load_pretrained_model
from llava.mm_utils import process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from PIL import Image
import requests
import copy
import torch
import warnings

warnings.filterwarnings("ignore")

pretrained = "AI-Safeguard/Ivy-VL-llava"

model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map)  # Add any other thing you want to pass in llava_model_args

model.eval()

# load image from url
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

# load image from local environment
# url = "./local_image.jpg"
# image = Image.open(url)

image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]

conv_template = "qwen_1_5"  # Make sure you use correct chat template for different models
question = DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image?"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()

input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
image_sizes = [image.size]

cont = model.generate(
    input_ids,
    images=image_tensor,
    image_sizes=image_sizes,
    do_sample=False,
    temperature=0,
    max_new_tokens=4096,
)

text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)

print(text_outputs)

Future Plan:

• We plan to release more versions of LLMs in different sizes.

• We will focus on improving the performance of the video modality.

Contact:

Feel free to contact us if you have any questions or suggestions📧:

• Email (Ivy Zhang): [email protected]

Citation:

If you find our work helpful, please consider citing our Model:

@misc{ivy2024ivy-vl,
    title={Ivy-VL:Compact Vision-Language Models Achieving SOTA with Optimal Data},
    url={https://huggingface.co/AI-Safeguard/Ivy-VL-llava},
    author={Ivy Zhang,Wei Peng,Jenny N,Theresa Yu and David Qiu},
    month={December},
    year={2024}
}