medical-image-captioning-roco

Medical Image Captioning Model

This model is fine-tuned for medical image captioning using the ROCOv2 radiology dataset.

Model Description

• Base Model: microsoft/git-base
• Training Data: ROCOv2 radiology images
• Task: Generate descriptive captions for medical/radiology images
• Language: English

Usage

from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image

# Load model and processor
processor = AutoProcessor.from_pretrained("WafaaFraih/medical-image-captioning-roco")
model = AutoModelForCausalLM.from_pretrained("WafaaFraih/medical-image-captioning-roco")

# Load and process image
image = Image.open("path_to_medical_image.jpg")
inputs = processor(images=image, return_tensors="pt")

# Generate caption
generated_ids = model.generate(pixel_values=inputs.pixel_values, max_length=100)
caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(caption)

Training Details

• Training Samples: 1800
• Validation Samples: 200
• Epochs: 10
• Batch Size: 8
• Learning Rate: 5e-05

Evaluation

Evaluated using Word Error Rate (WER) metric on medical image descriptions.

Limitations

• Trained on a subset of ROCOv2 dataset
• Performance may vary on different imaging modalities
• Should not be used for clinical diagnosis without expert validation

Citation

If you use this model, please cite:

@misc{medical-image-captioning,
  author = {WafaaFraih},
  title = {Medical Image Captioning Model},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/WafaaFraih/medical-image-captioning-roco}
}

Acknowledgments

• Base model: Microsoft GIT
• Dataset: ROCOv2 Radiology Dataset