MiniCPM-V-2_6-gqa-it-finetuned

<h1>Finetuned version of MiniCPM-V 2.6 on GQA-it</h1>

This is a fine-tuned version of MiniCPM-V 2.6 on GQA-it, designed for Italian Vision Question Answering. The original model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters.

Usage

You can visit the original basic model repository for advanced usage: https://github.com/OpenBMB/MiniCPM-V.

For more details about dataset please visit: https://github.com/crux82/gqa-it

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer,AutoProcessor

model = AutoModel.from_pretrained('sag-uniroma2/MiniCPM-V-2_6-gqa-it-finetuned', trust_remote_code=True,
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)
img="n346247.jpg"
image = Image.open(img).convert('RGB')

question = "C'è un idrante sull'erba?"
msgs = [{'role': 'user', 'content': [image,question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

GQA-it

Italian Question Answering on Image Scene Graphs

GQA-it is a large-scale Italian dataset for Visual Question Answering based on the balanced version of GQA.

GQA-it contains more than 1 million question/answer pairs in Italian over 80K images obtained by applying Neural Machine Translation.

Most importantly, a Test set of 3,000 question-answer pairs has been manually validated to provide a valuable benchmark in Italian.

Example

Citation

@inproceedings{DBLP:conf/nl4ai/ScaiellaMHC024,
  author={Antonio Scaiella and Daniele Margiotta and Claudiu Daniel Hromei and Danilo Croce and Roberto Basili},
  title={Evaluating Multimodal Large Language Models for Visual Question-Answering in Italian},
  year={2024},
  cdate={1704067200000},
  url={https://ceur-ws.org/Vol-3877/paper8.pdf},
  booktitle={NL4AI@AI*IA},
  crossref={conf/nl4ai/2024}
}