Downloads · 30 days
18
19% of all-time downloads
SophiaSirko/V-GIFT_llava_v1.5_qwen2.5_7B
V-GIFT_llava_v1.5_qwen2.5_7B is a image-text-to-text model from SophiaSirko. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
V-GIFT (Visually Grounded Instruction Fine-Tuning) checkpoint of LLaVA-1.5 with a Qwen2.5-7B-Instruct language model, from the NeurIPS 2026 paper V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance.
Downloads · 30 days
18
19% of all-time downloads
All-time downloads
93
Public
Repo size
3.3 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3.3 GB · 100%
From the Hugging Face model README
V-GIFT (Visually Grounded Instruction Fine-Tuning) checkpoint of LLaVA-1.5 with a Qwen2.5-7B-Instruct language model, from the NeurIPS 2026 paper V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance.
📄 Paper · 💻 Code · 🤗 Vicuna-7B variant
Multimodal LLMs often under-use visual information during instruction tuning, because many training examples can be solved from language priors alone. V-GIFT augments standard visual instruction tuning with a small fraction of visually grounded self-supervised tasks, phrased as ordinary instruction–response pairs:
These tasks cannot be solved without looking at the image. They need no human annotation, no architectural change and no extra training stage. Only the training data mix changes.
This checkpoint follows the LLaVA-1.5 recipe (CLIP ViT-L/14-336 vision encoder, MLP projector) with Qwen2.5-7B-Instruct as the language model, with SSL instructions mixed into LLaVA-v1.5-mix665k at a ratio ρ = 10%.
The checkpoint uses the LLaVA codebase with a Qwen2 language model (llava_qwen2), included in the V-GIFT repository:
git clone https://github.com/sirkosophia/V-GIFT.git
See the repository for training and evaluation instructions.
@misc{sirkogalouchenko2026boostingvisualinstructiontuning,
title={Boosting Visual Instruction Tuning with Self-Supervised Guidance},
author={Sophia Sirko-Galouchenko and Monika Wysoczanska and Andrei Bursuc and Nicolas Thome and Spyros Gidaris},
year={2026},
eprint={2604.12966},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2604.12966},
}
Built on LLaVA. Evaluation uses VLMEvalKit.