gemma3-alzheimer-lora

Gemma-3 4B VLM — Alzheimer MRI (QLoRA fine-tuned)

This repository contains LoRA-adapter weights for google/gemma-3-4b-it, fine-tuned on the Falah/Alzheimer_MRI dataset for single-image Alzheimer’s stage classification from MRI scans.

⚠️ Research/educational use only — not for clinical or diagnostic purposes.

---

Model Details

• Base model: google/gemma-3-4b-it
• Model type: Vision-Language model (VLM) with instruction-tuning
• Fine-tuning method: QLoRA (4-bit quantization, bf16 compute)
• Task: Image classification framed as a chat task (system+user prompts with image placeholder)
• Languages: English (prompting/instructions)
• Developed by: @shayanyan

Dataset

• Source: Falah/Alzheimer_MRI
• Description: MRI brain scans labeled into 4 Alzheimer stages:
  • Mild_Demented
  • Moderate_Demented
  • Non_Demented
  • Very_Mild_Demented
• Train size: ~5120 images
• Test size: ~1280 images

---

Uses

Direct Use

• Research into vision-language fine-tuning techniques
• Educational demos of QLoRA with Gemma-3

Out-of-Scope

• Medical diagnosis or treatment decisions
• Deployment in clinical workflows

---

Training Details

Preprocessing

• Converted dataset entries to chat format with image placeholders ({"type":"image"} in the user turn).
• Images resized/converted to RGB internally by the processor.
• Tokenized with seq_len=512.

Training Hyperparameters

• Batch size: 1 (micro-batch)
• Gradient accumulation: 16 → effective batch size ≈ 16
• Epochs: 1 (default demo run; recommend 3–5 for better minority-class coverage)
• Learning rate: 1.5e-4 (cosine schedule + 3% warmup recommended)
• Precision: bfloat16 compute, 4-bit quantized base weights
• Optimizer: AdamW (default in Trainer)
• LoRA config: rank=16, alpha=32, dropout=0.05; targets = [q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj]

Compute

• Hardware: NVIDIA L4 (Colab Pro)
• Runtime per epoch: ~10–15 minutes (with QLoRA, seq_len=512, effective batch=16)

---

Evaluation

• Metrics: Accuracy, Precision/Recall/F1, Confusion Matrix
• Results (subset, 64 train samples / 128 test samples):
  • High accuracy on Non_Demented (majority class)
  • Lower recall on minority classes (Moderate_Demented, Very_Mild_Demented)
  • Overall accuracy ~65% after 1 epoch (imbalanced)

Improvement expected with class balancing, more epochs, or weighted loss.

---

Environmental Impact

• Hardware Type: NVIDIA A100
• Cloud Provider: GKE
• Hours used: ~0.5 per epoch
• Emissions: Low (single GPU, <1 hr runtime for demo runs)

---

How to Use

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from PIL import Image

BASE = "google/gemma-3-4b-it"
REPO = "shayanyan/gemma3-alzheimer-lora"

processor = AutoProcessor.from_pretrained(REPO, trust_remote_code=True)
base = AutoModelForImageTextToText.from_pretrained(BASE, device_map="auto", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, REPO)

img = Image.open("example_mri.jpg").convert("RGB")
messages = [
    {"role":"system","content":[{"type":"text","text":"You are a careful neuroradiology assistant."}]},
    {"role":"user","content":[{"type":"text","text":"Classify the Alzheimer stage for this MRI."},{"type":"image"}]},
]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt")
inputs = {**inputs, "images": [[img]]}
outputs = model.generate(**{k:v.to(model.device) for k,v in inputs.items()}, max_new_tokens=32)
print(processor.tokenizer.decode(outputs[0], skip_special_tokens=True))