radtriage-ai-medgemma-4b

RadTriage AI — MedGemma 4B for Radiology Triage & Report Drafting

A prompt-engineered pipeline built on top of MedGemma 4B-IT from Google's Health AI Developer Foundations (HAI-DEF) for multimodal radiology triage and report generation.

Model Description

RadTriage AI uses MedGemma 4B-IT in a two-pass architecture designed to reduce hallucination in medical image interpretation:

• Pass 1 (Vision+Text): Extracts structured findings as JSON with anatomical location, severity, and calibrated confidence scores
• Pass 2 (Text-only): Generates a narrative radiology report grounded in Pass 1 findings — cannot hallucinate findings not detected in Pass 1
• Triage: Classifies urgency as CRITICAL / URGENT / ROUTINE with rationale

Base Model

This model traces to google/medgemma-4b-it — a multimodal medical AI model from Google's HAI-DEF collection, specifically designed for medical image interpretation.

How to Use

import torch
import json
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

# Load the base MedGemma model
model_id = "google/medgemma-4b-it"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id)

# Load the RadTriage prompt configuration
# (See radtriage_config.json for full prompt templates)

image = Image.open("chest_xray.png").convert("RGB")

# Pass 1: Structured Findings
pass1_prompt = """You are an expert radiologist assistant analyzing a Chest X-ray image.

Clinical indication: Shortness of breath
Prior studies: None

Specifically evaluate for: pneumothorax, pleural effusion, consolidation, 
cardiomegaly, mediastinal widening, rib fractures, pulmonary edema, masses, 
nodules, and lines/tubes.

Analyze the image systematically and respond ONLY with a valid JSON object 
following this exact schema:
{
  "modality_detected": "<specific modality>",
  "anatomical_region": "<body region>",
  "image_quality": "<adequate|suboptimal|non-diagnostic>",
  "quality_issues": [],
  "comparison_available": false,
  "findings": [
    {"id": 1, "description": "<finding>", "location": "<location>", 
     "severity": "<normal|mild|moderate|severe>", "confidence": 0.0}
  ],
  "overall_impression_category": "<normal|abnormal|critical>",
  "missing_context": []
}"""

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": pass1_prompt}
]}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)

input_len = inputs["input_ids"].shape[-1]
result = processor.decode(output_ids[0][input_len:], skip_special_tokens=True)
findings = json.loads(result)

# Pass 2: Narrative Report (text-only, no image)
pass2_prompt = f"""You are a radiologist drafting a formal report. Convert the 
following structured findings into a professional radiology report following 
ACR reporting guidelines.

STRUCTURED FINDINGS:
{json.dumps(findings, indent=2)}

Generate sections: EXAMINATION, CLINICAL INDICATION, COMPARISON, FINDINGS, IMPRESSION.
ONLY describe findings from the structured data. Do NOT invent new findings."""

messages_p2 = [{"role": "user", "content": [{"type": "text", "text": pass2_prompt}]}]
# ... generate report

Pipeline Configuration

The radtriage_config.json file contains:

• Modality-aware prompt templates for 5 imaging types
• Triage classification rules and critical keywords
• Confidence calibration thresholds
• Missing context checklist templates

Supported Modalities

Evaluation Results

Tested on 3 real NIH Chest X-ray cases:

Key Innovation: Two-Pass Anti-Hallucination

Traditional single-pass approaches generate reports directly from images, which can hallucinate findings. Our two-pass approach:

1. Pass 1 generates structured JSON (auditable, machine-readable)
2. Pass 2 generates prose ONLY from the JSON (cannot invent findings)

This architectural constraint is a safety feature for clinical AI applications.

Links

• Competition: MedGemma Impact Challenge
• Kaggle Notebook: RadTriage AI Pipeline
• GitHub: github.com/pramodmisra/radtriage-ai
• Live Demo: HuggingFace Spaces

Citation

@software{radtriage_ai_2026,
  title={RadTriage AI: Multimodal Radiology Triage & Report Drafting},
  author={Pramod Misra},
  year={2026},
  url={https://github.com/pramodmisra/radtriage-ai}
}

Disclaimer

⚠️ For demonstration and research purposes only. Not intended for clinical use, diagnosis, or treatment decisions. All medical image interpretations should be performed by qualified healthcare professionals.

Use of MedGemma is subject to the HAI-DEF Terms of Use.