medgemma-ae-detection

MedGemma AE Detection

MedGemma 4B fine-tuned to grade visible adverse events (AEs) from patient photographs. Part of CLARA, NEWMES AI team's submission to the MedGemma Impact Challenge (Kaggle, Feb 2026).

Results

Held-out test set: 42 synthetic images (2 per class). Exact-match accuracy on the 21-class label (AE type and grade).

Remaining errors are mostly off-by-one grade (e.g. pruritus G3 → G2) or normal ↔ grade 1 confusion. The full comparison, per-AE detection rates and a before/after gallery are in notebooks/2_medgemma+medsiglip+HeAR_SAE-detection.ipynb.

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "newmes/medgemma-ae-detection"
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

baseline = Image.open("baseline.jpg")   # pre-treatment photo
current  = Image.open("today.jpg")      # today's check-in frame

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": baseline},
        {"type": "image", "image": current},
        {"type": "text", "text":
         "Image 1 is the patient's baseline photo, Image 2 was taken today. "
         "Identify any new adverse event visible in Image 2 that is NOT present in Image 1 "
         "and grade it with CTCAE v5. Answer with one of: normal, or <ae_type> grade <1-3>. "
         "AE types: maculopapular rash, acneiform rash, periorbital edema, SJS prodrome, "
         "stomatitis, pruritus, alopecia."},
    ],
}]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True,
                                       tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model also works with a single image; the two-image prompt is how it is used inside CLARA. Recommended: bf16, one 24 GB GPU; served in CLARA via vLLM.

Intended use and limitations

• Research / demonstration only. Not a medical device, not clinically validated, not for diagnostic use.
• Trained and tested exclusively on synthetic (Gemini-generated) images — performance on real patient photos is unknown.
• Covers 7 dermatologic/visible AE types only; anything else is out of distribution.
• Tiny test set (42 images); accuracy figures have wide confidence intervals.
• Inherits the Health AI Developer Foundations terms of use from MedGemma.

About CLARA

CLARA (Clinical Longitudinal AI Research Assistant) tackles the 12–21-day blind spot between oncology-trial clinic visits: a daily ~60-second video + voice check-in (Data Collection Agent, on-device) feeds a longitudinal timeline with AE/SAE flagging and automatic MedWatch 3500A / E2B(R3) reports (Data Analysis Agent). A rule-based clinical-trial simulator (100 patients × 126 days, 7 real drug profiles) is used to quantify the benefit: mean AE detection delay 4.6 → 1.2 days, deaths 21 → 16 per 100 patients.

HAI-DEF models used: MedGemma 1.5 4B (nurse dialogue, visual AE grading), MedSigLIP-448 (frame classification), HeAR (dry/wet cough), MedASR (medical speech recognition).

• Code: newmes/clara-web · iOS app: newmes/clara-app
• Sibling model: newmes/medgemma-4b-antihallu · data: newmes/clinical-trial-engine-data
• Competition: The MedGemma Impact Challenge — Google Research / HAI-DEF, $100 000 prize pool, final submission 2026-02-24, 850+ teams. Judged on effective use of HAI-DEF models, problem importance, real-world impact, technical feasibility, and execution.

Originally published as AlphaRaven/medgemma-ae-detection (2026-02-23); moved to the newmes organization in Sep 2026. Weights are identical.

Citation

@misc{clara2026,
  title  = {CLARA: Clinical Longitudinal AI Research Assistant},
  author = {NEWMES AI Team},
  year   = {2026},
  url    = {https://github.com/newmes/clara-web},
  note   = {MedGemma Impact Challenge submission}
}