llava-llama3-docci

Fine tuned version of xtuner/llava-llama-3-8b-v1_1 on gokaygokay/random_instruct_docci dataset.

pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps
pip install lmdeploy

# Google Colab Error Fix
import nest_asyncio
nest_asyncio.apply()

from lmdeploy import pipeline
from lmdeploy.vl import load_image

pipe = pipeline('gokaygokay/llava-llama3-docci')

image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
response = pipe(('describe this image', image))
print(response)

Academic use and citation

This model is evaluated as a dense-captioning baseline in Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (EMNLP 2024).

If this model is useful in your research, please cite both the model and the paper that documents its academic use:

@software{aydogan2024llava_llama3_docci,
  author = {Aydoğan, Gökay},
  title = {LLaVA-Llama-3-DOCCI},
  year = {2024},
  url = {https://huggingface.co/gokaygokay/llava-llama3-docci},
  note = {Hugging Face model}
}

@inproceedings{chung2024selective,
  author = {Chung, Jiwan and Lee, Sungjae and Kim, Minseo and Han, Seungju and Yousefpour, Ashkan and Hessel, Jack and Yu, Youngjae},
  title = {Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  pages = {2423--2451},
  doi = {10.18653/v1/2024.emnlp-main.143}
}

Maintainer: Gökay Aydoğan