blip-yoda-captioning

Image captioning model finetuned on BLIP-base, responds like how Yoda speaks,

"Sitting in a car, a man is"

Try web app here: https://yodacaptioner.up.railway.app/

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

An image-to-text model finetuned on BLIP-base with the transformers package

• Developed by: vkao8264
• Model type: Image-to-text
• Language(s) (NLP): English
• License: bsd-3-clause
• Finetuned from model [optional]: blip-image-captioning-base

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

from PIL import Image
from transformers import AutoProcessor, BlipForConditionalGeneration

processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("vkao8264/blip-yoda-captioning")

filepath = "path-to-your-image"
raw_image = Image.open(filepath).convert('RGB')

inputs = processor(raw_image, return_tensors="pt").to("cuda")
output_tokens = model.generate(**inputs)
caption = processor.decode(output_tokens[0], skip_special_tokens=True)
print(caption)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was fine-tuned on 30000 image-caption pairs from the COCO captions dataset. Specifically, captions_train2014.

Before training, captions were changed to yoda-style captions using phi3 with few-shot learning

Scripts can be found on https://github.com/vincent8264/yoda_captioning