pcamvlm-medgemma-lora

PcamVLM v2 — MedGemma-1.5-4B QLoRA Adapter for Histopathology

A QLoRA adapter (LoRA r=16, α=32, ~33M trainable params) fine-tuned on top of google/medgemma-1.5-4b-it for two real histopathology tasks on a single model:

1. PCam (PatchCamelyon) — binary tumor/normal classification of H&E sentinel lymph-node patches.
2. NCT-CRC-HE-100K — 9-way colorectal tissue typing (ADI/BACK/DEB/LYM/ MUC/MUS/NORM/STR/TUM).

Adapter only — the base model is the gated MedGemma 1.5 4B (Google, Gemma Terms of Use). It is not re-hosted here. Research/education use only — not a clinical device.

Evaluation (real, held-out data — best checkpoint step 1750)

All numbers were produced by eval_scientific.py on the best-eval-loss training checkpoint (step 1750, eval loss 0.0776), base in 4-bit NF4.

✅ PCam — binary tumor/normal

Balanced n=3000 from the held-out test split (mirror: zacharielegault/PatchCamelyon). Method: logit-level yes/no comparison at the final prompt position (no sampling, no free-text parsing).

✅ NCT-CRC — 9-way tissue typing

Balanced n=900 from the official held-out CRC_VAL_HE_7K split (mirror: 1aurent/NCT-CRC-HE). Method: greedy free-text decoding + label parsing.

Per-class F1 (support 100 each): BACK 0.985 · NORM 0.943 · ADI 0.936 · DEB 0.921 · MUC 0.930 · TUM 0.846 · LYM 0.840 · MUS 0.762 · STR 0.717.

What changed vs v1

v1 was fine-tuned with synthetic NCT-CRC distractors; real-data evaluation measured 5.8% (chance-level) on CRC_VAL_HE_7K. v2 trains the 9-way head on real, balanced NCT-CRC data instead, lifting held-out accuracy from 5.8% → 87.6% while PCam holds steady (90.4% → 90.8%).

Training

• Data (all real, no synthetic):
  • PCam: 13,333 patches from the train split (binary tumor/normal).
  • NCT-CRC: 3,954 balanced patches (≈440/class, all 9 classes) from a mini-subset of the NCT-CRC-HE-100K training split (junyeong-nero/ mini-NCTCRCHE100K), verified disjoint from the CRC_VAL_HE_7K eval set by image-byte hashing (overlap = 0).
• Method: QLoRA 4-bit NF4 (double-quant), bf16 compute, 1 epoch, batch 1 × grad-accum 8 (effective batch 8), lr 2e-4 cosine, warmup 3%, max length 512, gradient checkpointing, LoRA on all attention + MLP projections.
• Hardware: single RTX 3090 (24 GB, ≈8 GB used), ~4 h.
• Runtime fix: 224×224 inputs (vs the SigLIP tower's native 896×896) require interpolated position embeddings + adaptive projector pooling — applied at load time by src/model.py:patch_siglip_interpolation.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"
processor = AutoProcessor.from_pretrained(base, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "vivekkopthsd/pcamvlm-medgemma-lora")
model.eval()

# PCam binary
image = Image.open("patch.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Does the center region of this H&E stained "
                             "lymph node patch contain tumor tissue? "
                             "Answer yes or no."},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=10)
print(processor.decode(out[0], skip_special_tokens=True))

For the 9-way NCT task, ask "Which tissue type is shown? Reply with exactly one label from ADI, BACK, DEB, LYM, MUC, MUS, NORM, STR, TUM."

Reproducibility

• Full eval script + results: eval_scientific.py, models/eval_metrics.json.
• Evaluation seeds: PCam 42, NCT-CRC 7; eval sets disjoint from training data.
• Adapter provenance: models/pcam-medgemma-lora-nct/training_metadata.json.

Citation

@misc{pcamvlm-medgemma-2026,
  title={PcamVLM v2: QLoRA fine-tuning of MedGemma-1.5-4B for PCam and real-data NCT-CRC histopathology classification},
  author={Kopu, Vivek},
  year={2026},
  howpublished={\url{https://huggingface.co/vivekkopthsd/pcamvlm-medgemma-lora}}
}