Downloads · 30 days
38
4% of all-time downloads
VLM2Vec-LLaVa-Next-vllm
parasail-ai/VLM2Vec-LLaVa-Next-vllm
VLM2Vec-LLaVa-Next-vllm is a image-text-to-text model from parasail-ai. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
A new checkpoint trained using llava-v1.6-mistral-7b-hf with an enhanced training setup (LoRA tuning, batch size of 2048, maximum sub-dataset size of 100k). This model has shown significantly improved performance on M…
All-time downloads
856
Public
Parameters
7.6B
15.1 GB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.safetensors15.1 GB · 100%
At a glance
- Task
- Image-Text-to-Text
- Library
- transformers
- License
- apache-2.0
- Model type
- llava_next
- Access
- Public
- Created
- Dec 11, 2024
- Updated
- Dec 11, 2024
- SHA
- 7811d408
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
A new checkpoint trained using llava-v1.6-mistral-7b-hf with an enhanced training setup (LoRA tuning, batch size of 2048, maximum sub-dataset size of 100k). This model has shown significantly improved performance on MMEB & Flickr30K compared to the previous Phi-3.5-based model.
This repo contains the code and data for VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. In this paper, we focus on building a unified multimodal embedding model suitable for a wide range of tasks. Our approach is based on transforming an existing, well-trained Vision-Language Model (VLM) into an embedding model. The core idea is to append an [EOS] token at the end of the input sequence, which serves as the representation for the combined multimodal inputs.
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- llava_next
- License
- apache-2.0
- Languages
- en
- Created
- Dec 11, 2024
- Updated
- Dec 11, 2024