gemma-4-E2B-it-ham10000-lora

Gemma 4 4B-it fine-tuned on HAM10000 (LoRA)

LoRA adapter fine-tuned from google/gemma-4-E2B-it on marmal88/skin_cancer for 7-class dermoscopic lesion classification.

Labels

• nv
• mel
• bkl
• akiec
• bcc
• df
• vasc

Eval (HAM10000 test split)

• Accuracy: 0.6300
• Macro F1: 0.2537
• Unknown outputs: 0/200

Full per-class metrics are in the attached metrics.json.

Usage

from peft import PeftModel
from transformers import AutoProcessor

try:
    from transformers import AutoModelForMultimodalLM as _AutoModelForMM
except ImportError:
    from transformers import AutoModelForImageTextToText as _AutoModelForMM  # pragma: no cover
import torch

BASE = "google/gemma-4-E2B-it"
ADAPTER = "mufasabrownie/gemma-4-E2B-it-ham10000-lora"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForMultimodalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)  # or the auto class matching your transformers version
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "<PIL.Image or URL>"},
        {"type": "text", "text": "Classify this dermoscopic image into one of: "
                                 "nv, mel, bkl, akiec, bcc, df, vasc. Answer with the label only."},
    ],
}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=your_image, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training

• Method: LoRA (r=16, alpha=32, dropout=0.05) on attention + MLP projections
• Precision: bf16
• Optimizer: AdamW, cosine schedule, lr=2e-4, 10% warmup
• Epochs: 3
• Class imbalance: weighted random sampler + class-weighted cross-entropy