llava-dinov2-internlm2-7b-v1

Model

llava-dinov2-internlm2-7b-v1 is a LLaVA model fine-tuned from InternLM2-Chat-7B and Dinov2-large with LLaVA-Pretrain and LLaVA-Instruct by XTuner. I thank the help of Zhihao Lin and pppppM from the Xtuner team. I also thank the Huggingface transformers team for approving my pull request so training Dinov2 in bf16 becomes possible.

I did not carefully tune the training hyperparameters but the model still show capability to solve some tasks. It shows that a visual encoder can be integrated with an LLM, even when the encoder is not aligned with natural language with contrastive learning like CLIP.

Future development of Dinov2 based LLaVA

Using Dinov2 as the vision encoder of LLaVA may have some disadvantages. Unlike CLIP, Dinov2 is not pre-aligned with language embedding space. Even if you use both CLIP and Dinov2 and mix their tokens, the benchmark perfermance is not very strong (see arxiv:2401.06209 and the following table from their paper).

If you have any idea to improve it, please open an issue or just send an email to [email protected]. You are welcomed!

Example

Explain the photo in English: Explain the photo in Chinese:

Rank

Results

Installation

git clone https://github.com/InternLM/xtuner
pip install -e ./xtuner[deepspeed]
apt install git-lfs
cd ./xtuner
# Now replace the source code files with the modifed version in modified_xtuner_code directory

Chat

xtuner chat internlm/internlm2-chat-7b \
--visual-encoder facebook/dinov2-large\
--llava ./lora_and_projector \
--prompt-template internlm2_chat \
--image $IMAGE_PATH

Common Errors

command error: 'libGL.so.1: cannot open shared object file: No such file or directory'!

You can solve it by

# For Ubuntu
sudo apt-get update
sudo apt-get install libgl1-mesa-glx

# For CentOS and Fedora
sudo yum install mesa-libGL

Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
        Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.

You can solve it by reinstall numpy.

ImportError: 
InternLM2Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the

You just need

pip install protobuf

To use tensorboard to visualize the training loss curve:

pip install future tensorboard 

1. If your training process is killed during data preprocessing, you can modify the map_num_proc in xtuner/xtuner/dataset /huggingface.py

def process(dataset,
            do_dataset_tokenization=True,
            tokenizer=None,
            max_length=None,
            dataset_map_fn=None,
            template_map_fn=None,
            max_dataset_length=None,
            split='train',
            remove_unused_columns=False,
            rename_maps=[],
            shuffle_before_pack=True,
            pack_to_max_length=True,
            use_varlen_attn=False,
            input_ids_with_output=True,
            with_image_token=False,
            map_num_proc=32): # modify it to a smaller number, e.g., 4

1. If you fail to load the model, check whether you installed git-lfs and actually downloaded the model file.

Data prepration

1. File structure

# . means the llava-dinov2-internlm2-7b-v1 folder you clone
./data/llava_data
├── LLaVA-Pretrain
│   ├── blip_laion_cc_sbu_558k.json
│   ├── blip_laion_cc_sbu_558k_meta.json
│   └── images
├── LLaVA-Instruct-150K
│   └── llava_v1_5_mix665k.json
└── llava_images
    ├── coco
    │   └── train2017
    ├── gqa
    │   └── images
    ├── ocr_vqa
    │   └── images
    ├── textvqa
    │   └── train_images
    └── vg
        ├── VG_100K
        └── VG_100K_2

1. Pretrain Data

LLaVA-Pretrain

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain --depth=1

1. Finetune Data

3.1 Text data

LLaVA-Instruct-150K

  ```shell
  # Make sure you have git-lfs installed (https://git-lfs.com)
  git lfs install
  git clone https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K --depth=1
  ```

3.2 Image data

3.2.1 COCO (coco): train2017

3.2.2 GQA (gqa): images

3.2.3 OCR-VQA (ocr_vqa): download script

  ⚠️⚠️⚠️ Modify the name of OCR-VQA's images to keep the extension as `.jpg`!

     ```shell
     #!/bin/bash
     ocr_vqa_path="<your-directory-path>"

     find "$target_dir" -type f | while read file; do
         extension="${file##*.}"
         if [ "$extension" != "jpg" ]
         then
             cp -- "$file" "${file%.*}.jpg"
         fi
     done
     ```

3.2.4 TextVQA (textvqa): train_val_images

3.2.5 VisualGenome (VG): part1, part2

Cheers! Now train your own model!

1. Alignment module pretraining

NPROC_PER_NODE=8 xtuner train ./llava_internlm2_chat_7b_dinov2_e1_gpu8_pretrain.py --deepspeed deepspeed_zero2

Remember to change the batch size and gradient accumulation parameters to fit your hardware. So your GPU_num * batch_size * gradient_accumulation is roughly equal to mine to reproduce the result.

The checkpoint and tensorboard logs are saved by default in ./work_dirs/. I only train it for 1 epoch to be same as the original LLaVA paper. Some researches also report that training for multiple epochs will make the model overfit the training dataset and perform worse in other domains.

Here is my loss curve:

1. Instruction following fine-tuning

NPROC_PER_NODE=8 xtuner train ./llava_internlm2_chat_7b_dinov2_e1_gpu8_finetune.py --deepspeed deepspeed_zero2

Here is my loss curve (the curve fluctuates strongly because the batch size is small, and I only record batch loss instead of epoch loss):

Transfer the checkpoints to Huggingface safetensor format

xtuner convert pth_to_hf ./llava_internlm2_chat_7b_dinov2_e1_gpu8_finetune.py ./work_dirs/epoch_1.pth ./my_lora_and_projector

The adapter still need to be used with the internlm/internlm2-chat-7b and facebook/dinov2-large models. I have not tried to merge them yet but it is possible with Xtuner, see this tutorial.

MMBench Evaluation

You can first download the MMBench data:

wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_EN.tsv
wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_EN.tsv
wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_CN.tsv
wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_CN.tsv
wget https://opencompass.openxlab.space/utils/VLMEval/CCBench.tsv

Then run:

NPROC_PER_NODE=8 xtuner mmbench internlm/internlm2-chat-7b \
--visual-encoder facebook/dinov2-large \
--llava ./my_lora_and_projector \
--prompt-template internlm2_chat \
--data-path $MMBENCH_DATA_PATH \
--work-dir $RESULT_PATH

You can also use VLMEvalKit to evaluate it on other benckmarks.

Deployment

Xtuner team is developing HF chatbot (based on Huggingface transformers) and LMDeploy chatbot (based on TurboMind). I am waiting for their final version of API.