ALLaVA-3B-Longer

ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model

this is a fork from the official repo

<p align="center"> ⚡ALLaVA is a project that provides a large-scale GPT4V-synthesized dataset for training LVLMs.⚡ </p> <!-- <p align="center">    </p> --> <p align="center"> 📃 <a href="https://arxiv.org/abs/2402.11684" target="_blank">Paper</a> • 🌐 <a href="https://allava.freedomai.cn/#/" target="_blank">Demo</a> • 👨🏻‍💻 <a href="https://github.com/FreedomIntelligence/ALLaVA" target="_blank">Github</a> </p> <p align="center"> 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V" target="_blank">ALLaVA-4V Dataset</a> </p> <p align="center"> 🤗 <a href="https://huggingface.co/FreedomIntelligence/ALLaVA-3B-Longer" target="_blank">ALLaVA-3B-Longer</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/ALLaVA-3B" target="_blank">ALLaVA-3B</a> </p> <!-- <p align="center"> 📃 <a href="https://arxiv.org/abs/2402.11684" target="_blank">Paper</a> • 🌐 <a href="https://allava.freedomai.cn/#/" target="_blank">Demo</a> • 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V" target="_blank">ALLaVA-4V Dataset</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/ALLaVA-3B-Longer" target="_blank">ALLaVA-3B-Longer</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/ALLaVA-3B" target="_blank">ALLaVA-3B</a> <br> <a href="https://github.com/FreedomIntelligence/CMB/blob/main/README_zh.md"> 中文</a> | <a href="https://github.com/FreedomIntelligence/CMB/blob/main/README.md"> English </p> -->

Benchmark Result

Our model ALLaVA-3B-Longer and ALLaVA-3B achieve competitive results on 12 benchmarks. Bold numbers denote the SOTA performance among 3B-scale models.

The detailed information of each benchmark is shown in Table 4 of our technical report.

🏭 Inference

Load from 🤗 (Recommended)

See the example script.

CLI

See here for CLI code snippet.

🏋️‍♂️ Training

Data

<div align=center> <img src="training_datasets_by_stage.jpg" width = "640" alt="training_datasets" align=center /> </div>

As shown in the table, ALLaVA-3B uses 1M and 1.5M data for PT. and FT., respectively. ALLaVA-3B-Longer trains one more epoch (i.e. 3M in total) for the FT. stage.

Code

The training code is largely based on LLaVA-v1.5. We wholeheartedly express our gratitude for their invaluable contributions to open-sourcing LVLMs.

Cost

We train our models on 8*A800 GPUs. ALLaVA-3B-Longer takes 8.3h for PT and 21.3h for FT. ALLaVA-3B takes 8.3h for PT and 10.6h for FT. These two models share the same PT procedure.

Hyperparameters

The LM backbone, projector are trainable, while the vision encoder is kept frozen. The trainabilities of each module are the same for both stages.

📚 ALLaVA-4V Data

The majority part of training data is ALLaVA-4V. See here to prepare it for training.

🙌 Contributors

• Project Leader: Guiming Hardy Chen

• Data: Shunian Chen, Junying Chen, Xiangbo Wu

• Evaluation: Ruifei Zhang

• Deployment: Xiangbo Wu, Zhiyi Zhang

• Advising: Zhihong Chen, Benyou Wang

• Others: Jianquan Li, Xiang Wan

📝 Citation

If you find our data useful, please consider citing our work! We are FreedomIntelligence from Shenzhen Research Institute of Big Data and The Chinese University of Hong Kong, Shenzhen

@article{chen2024allava,
  title={ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model},
  author={Chen, Guiming Hardy and Chen, Shunian and Zhang, Ruifei and Chen, Junying and Wu, Xiangbo and Zhang, Zhiyi and Chen, Zhihong and Li, Jianquan and Wan, Xiang and Wang, Benyou},
  journal={arXiv preprint arXiv:2402.11684},
  year={2024}
}