Downloads · 30 days
0
parthsalke/medclip-glioma-lora-8bit
medclip-glioma-lora-8bit is a zero-shot image classification model from parthsalke. Use it for the zero-shot image classification task on the model card, and read the license before you ship it in a product. It is set up for transformers.
A fine-tuned version of PubMedCLIP using LoRA adapters on the BRATS-derived Glioma dataset for highly precise brain tumor classification.
Downloads · 30 days
0
Access
Public
Updated May 5, 2025
Repo size
8.9 MB
Likes
0
Public
Click a slice to open those files.
.safetensors8.9 MB · 91%
From the Hugging Face model README
A fine-tuned version of PubMedCLIP using LoRA adapters on the BRATS-derived Glioma dataset for highly precise brain tumor classification.
This model builds upon pre-trained PubMedCLIP — a CLIP variant pre-trained on radiology images from the ROCO dataset — and adapts it for Glioma tumor classification using supervised fine-tuning (SFT) with Low-Rank Adaptation (LoRA) and 8-bit quantization.
flaviagiammarino/pubmed-clip-vit-base-patch32bitsandbytesparthsalke/vlm_glioma_dataset| Component | Strategy |
|---|---|
| Vision Encoder | Frozen base, LoRA adapters (qkv) |
| Text Encoder | Frozen base, LoRA adapters |
| Training Method | Supervised Fine-Tuning (SFT) |
| Optimizer | AdamW |
| LoRA Params | r=8, alpha=32, dropout=0.1 |
| Precision | 8-bit (bnb + PEFT) |
from transformers import CLIPProcessor
from peft import PeftModel
from PIL import Image
# Load base + LoRA adapter
from transformers import CLIPModel
base = CLIPModel.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")
model = PeftModel.from_pretrained(base, "parthsalke/biomedclip-glioma-lora-8bit")
# Load processor
processor = CLIPProcessor.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")
# Inference example
image = Image.open("example.png").convert("RGB")
text = [" ", " "] #Input labels here
inputs = processor(images=image, text=text, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1)