V-GIFT_llava_v1.5_vicuna7b

V-GIFT: LLaVA-1.5 Vicuna-7B

V-GIFT (Visually Grounded Instruction Fine-Tuning) checkpoint of LLaVA-1.5 with Vicuna-7B-v1.5, from the NeurIPS 2026 paper V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance.

📄 Paper · 💻 Code · 🤗 Qwen2.5-7B variant

Model description

Multimodal LLMs often under-use visual information during instruction tuning, because many training examples can be solved from language priors alone. V-GIFT augments standard visual instruction tuning with a small fraction of visually grounded self-supervised tasks, phrased as ordinary instruction–response pairs:

• Rotation prediction: predict the rotation applied to an image.
• Point-wise colorization: match points on a grayscale image to their true colors.
• Cross-view point correspondence: find corresponding points across two views.

These tasks cannot be solved without looking at the image. They need no human annotation, no architectural change and no extra training stage. Only the training data mix changes.

This checkpoint follows the LLaVA-1.5 recipe (CLIP ViT-L/14-336 vision encoder, MLP projector, Vicuna-7B-v1.5 language model), with SSL instructions mixed into LLaVA-v1.5-mix665k at a ratio ρ = 10%.

Usage

git clone https://github.com/sirkosophia/V-GIFT.git

See the repository for training and evaluation instructions.

Citation

@misc{sirkogalouchenko2026boostingvisualinstructiontuning,
      title={Boosting Visual Instruction Tuning with Self-Supervised Guidance},
      author={Sophia Sirko-Galouchenko and Monika Wysoczanska and Andrei Bursuc and Nicolas Thome and Spyros Gidaris},
      year={2026},
      eprint={2604.12966},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.12966},
}

Acknowledgements

Built on LLaVA. Evaluation uses VLMEvalKit.