Downloads · 30 days
15
12% of all-time downloads
jayantsom/medgemma-1v5-4b-it-rsna23-abd-ct-peft-lora-r16-a32-ep3-lr2e4-v1
medgemma-1v5-4b-it-rsna23-abd-ct-peft-lora-r16-a32-ep3-lr2e4-v1 is a image-text-to-text model from jayantsom. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as mit.
This is a Parameter-Efficient Fine-Tuning (LoRA) adapter for google/medgemma-1.5-4b-it. It specializes the base medical vision-language model to act as an expert trauma radiologist, analyzing abdominal CT angiogram vo…
Downloads · 30 days
15
12% of all-time downloads
All-time downloads
130
Public
Repo size
165 MB
Likes
2
Public
Click a slice to open those files.
.safetensors131 MB · 80%
From the Hugging Face model README
This is a Parameter-Efficient Fine-Tuning (LoRA) adapter for google/medgemma-1.5-4b-it. It specializes the base medical vision-language model to act as an expert trauma radiologist, analyzing abdominal CT angiogram volumes to detect and classify solid organ injuries and hemorrhages, outputting highly structured JSON clinical reports.
The base MedGemma 1.5 (4B parameters) model has been fine-tuned using LoRA on the RSNA 2023 Abdominal Trauma Detection dataset. Instead of open-ended conversational text, this adapter strictly aligns the model to evaluate multi-slice CT volumes and generate a structured JSON output detailing the injury pattern, specific organs involved (liver, spleen, kidney, bowel), bleeding description, severity estimation, and differential diagnoses.
google/medgemma-1.5-4b-itThis adapter is intended to be loaded on top of the base medgemma-1.5-4b-it model. It is designed to take an interleaved sequence of 2.5D CT slice images (NIfTI/DICOM converted to RGB via soft-tissue windowing) and output a precise JSON schema. It is a core component of the HAI-DEF multi-model trauma analysis pipeline.
[More Information Needed]
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Use the code below to get started with the model.
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
from peft import PeftModel
# 1. Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
base_model = AutoModelForImageTextToText.from_pretrained(
"google/medgemma-1.5-4b-it",
quantization_config=bnb_config,
device_map="auto"
)
processor = AutoProcessor.from_pretrained("google/medgemma-1.5-4b-it")
# 2. Load this LoRA adapter
model = PeftModel.from_pretrained(base_model, "jayantsom/medgemma-1v5-4b-it-rsna23-abd-ct-peft-lora-r16-a32-ep3-lr2e4-v1")
# 3. Prepare Image and Prompt
image = Image.open("path_to_ct_slice.png").convert("RGB")
messages = [{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "You are a trauma radiologist. Analyze this abdominal CT angiogram slice for hemorrhage and solid organ injury. Respond in JSON with keys: injury_pattern, organs_involved, bleeding_description, severity_estimate, differential_diagnosis."}
]
}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=800, do_sample=False) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
The model was fine-tuned on a curated 200-sample subset of the RSNA 2023 Abdominal Trauma Detection dataset (jherng/rsna-2023-abdominal-trauma-detection). Multi-slice 3D NIfTI volumes were lazily loaded into memory as streams to optimize processing.
CT NIfTI volumes were loaded and dynamically sliced using a 2.5D multi-slice approach. Slices were extracted from the middle 60% of the volume, and soft-tissue windowing (Center: 50 HU, Width: 400 HU) was applied to map the data into 3-channel RGB PIL Images.
bfloat16 mixed precision via BitsAndBytes NF4 4-bit quantization.sdpa (Scaled Dot Product Attention)paged_adamw_8bitq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_projEvaluation was performed on a hold-out validation split of 20 samples from the RSNA 2023 Abdominal Trauma dataset that the model did not see during the training phase.
Evaluation disaggregates performance based on:
severity_estimate field compared to radiologist ground-truth labels.The LoRA fine-tuning successfully shifted MedGemma from a conversational assistant into a strict clinical parser. It demonstrates high capability in recognizing massive hemorrhage and solid organ damage, though it occasionally struggles with subtle, low-grade bowel injuries due to the limited 200-sample dataset size.
Early qualitative examination indicates that the model heavily relies on the interleaved 2.5D visual tokens to identify active extravasation (bright contrast pooling). It demonstrates strong cross-modal alignment between the visual presence of hemoperitoneum and the generated bleeding_description text.
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
The underlying architecture is MedGemma 1.5 (a PaliGemma-style Multimodal Vision-Language Model). The objective is Autoregressive Causal Language Modeling conditioned on multimodal image-text tokens, specifically parameterized via Low-Rank Adaptation (LoRA) matrices applied to the attention and MLP layers.
BibTeX:
@misc{rsna2023trauma,
author = {RSNA},
title = {RSNA 2023 Abdominal Trauma Detection},
year = {2023},
publisher = {Kaggle},
url = {https://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection}
}
APA:
Radiological Society of North America (RSNA). (2023). RSNA 2023 abdominal trauma detection. Kaggle. https://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection
This adapter is released for educational and research purposes. I encourage the community to:
I am open to collaboration, questions, and feedback. Feel free to reach out via Hugging Face or email below.
Jayant Som
Jayant Som
(Reach out via Hugging Face profile or [email protected])