BLIP-finetune-on-COCO

<div align="center">

BLIP-LoRA Image Captioning Model

Fine-tuned BLIP for Image Captioning on COCO 2014

This repository contains a LoRA fine-tuned adapter for Salesforce/blip-image-captioning-base.

git hub repo: github

</div>

---

Model Overview

This model is based on BLIP Image Captioning Base and fine-tuned using LoRA.

Instead of storing the full BLIP model weights, this repository stores only the lightweight LoRA adapter weights. During inference, the base BLIP model is loaded first, and then this LoRA adapter is attached.

---

Repository Files

The repository should contain these files:

.
├── adapter_config.json
├── adapter_model.safetensors
├── preprocessor_config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── vocab.txt
├── training_state.pt
├── blip_lora_coco2014.ipynb
└── README.md

Important: make sure the files are named exactly like this:

adapter_config.json
adapter_model.safetensors
preprocessor_config.json
tokenizer.json
tokenizer_config.json
special_tokens_map.json
vocab.txt

Do not upload them with names like:

adapter_config(1).json
adapter_model(1).safetensors

because PEFT expects the standard file names.

---

LoRA Configuration

The adapter targets the attention projection layers of BLIP, allowing the model to adapt to COCO-style image captions while keeping most of the base model frozen.

---

Image Preprocessing

The model uses the BLIP image processor.

---

Training Details

---

Evaluation Results

Evaluation was performed on 200 validation images using beam search.

Example generated captions:

a bed with a book on top of it in front of a window
a police car parked on the side of the road
a man standing in front of a street sign
a baseball player swinging a bat at a ball

---

How to Use

Install the required libraries:

pip install transformers peft accelerate safetensors pillow torch

Then load the base BLIP model and attach the LoRA adapter:

import torch
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
from peft import PeftModel

repo_id = "your-username/your-model-repo"
base_model_id = "Salesforce/blip-image-captioning-base"

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained(repo_id)

base_model = BlipForConditionalGeneration.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, repo_id)
model = model.to(device)
model.eval()

Generate a caption:

image = Image.open("example.jpg").convert("RGB")

inputs = processor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        pixel_values=inputs["pixel_values"],
        max_length=30,
        num_beams=5
    )

caption = processor.decode(output[0], skip_special_tokens=True)
print(caption)

---

Optional: Merge LoRA Adapter

For deployment, you can merge the LoRA adapter into the base model:

merged_model = model.merge_and_unload()

After merging, the model becomes a standard BLIP model with the LoRA weights applied.

---

Intended Use

This model is intended for:

• Image caption generation
• Vision-language learning experiments
• COCO-style captioning tasks
• Educational fine-tuning demonstrations
• Lightweight BLIP adaptation using LoRA

---

Limitations

This model has several limitations:

• It is a LoRA adapter, not a full standalone model.
• The base model Salesforce/blip-image-captioning-base is required for inference.
• Caption quality depends on the COCO 2014 training subset.
• The model may generate generic captions for complex scenes.
• The model may hallucinate objects that are not visible in the image.
• The model may perform poorly on images outside the COCO-style distribution.
• The model is not designed for safety-critical use cases.

---

Out-of-Scope Use

This model is not intended for:

• Medical image interpretation
• Legal or forensic image analysis
• Safety-critical decision-making
• Surveillance or biometric identification
• Production use without further evaluation

---

Citation

@misc{blip_lora_coco2014_captioning,
  title = {BLIP-LoRA Image Captioning Model Fine-Tuned on COCO 2014},
  author = {Ali Sedghiye},
  year = {2026},
  note = {LoRA adapter for Salesforce/blip-image-captioning-base}
}

---

Author

Developed by Ali Sedghiye as a LoRA fine-tuned BLIP image captioning model.