paligemma-rich-captions

Fine tuned version of PaliGemma model on google/docci dataset with middle size captions between 200 and 350 characters. This model has less halucinations.

pip install git+https://github.com/huggingface/transformers

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "gokaygokay/paligemma-rich-captions"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

model = PaliGemmaForConditionalGeneration.from_pretrained(model_id).to('cuda').eval()
processor = AutoProcessor.from_pretrained(model_id)

## prefix
prompt = "caption en"
model_inputs = processor(text=prompt, images=image, return_tensors="pt").to('cuda')
input_len = model_inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**model_inputs, max_new_tokens=256, do_sample=False)
    generation = generation[0][input_len:]
    decoded = processor.decode(generation, skip_special_tokens=True)
    print(decoded)

Academic use and citation

This model is one of the image-captioning systems evaluated in Automated Interpretation of Non-Destructive Evaluation Contour Maps Using Large Language Models for Bridge Condition Assessment (IEEE BigData 2024).

If this model is useful in your research, please cite both the model and the paper that documents its academic use:

@software{aydogan2024paligemma_rich_captions,
  author = {Aydoğan, Gökay},
  title = {PaliGemma Rich Captions},
  year = {2024},
  url = {https://huggingface.co/gokaygokay/paligemma-rich-captions},
  note = {Hugging Face model}
}

@inproceedings{darji2024automated,
  author = {Darji, Viraj Nishesh and Liao, Callie C. and Liao, Duoduo},
  title = {Automated Interpretation of Non-Destructive Evaluation Contour Maps Using Large Language Models for Bridge Condition Assessment},
  booktitle = {2024 IEEE International Conference on Big Data (BigData)},
  year = {2024},
  pages = {3258--3263},
  doi = {10.1109/BigData62323.2024.10825532}
}

Maintainer: Gökay Aydoğan