ham10000-multimodal-skin-lesion

HAM10000 Multimodal Skin Lesion Classification

Cross-attention fusion model for 7-class dermoscopic skin lesion classification, combining image features with patient metadata (age, sex, lesion localization). Trained from scratch on HAM10000 using PyTorch.

Model description

The model uses a CNN-B backbone (residual connections + batch normalisation) as the image encoder, paired with a cross-attention fusion module that allows patient metadata to selectively attend to spatial regions of the image feature map. This produces a metadata-conditioned image representation rather than a spatially-averaged one.

Performance

Per-class AUROC

How to use

import torch
from PIL import Image
from torchvision import transforms

# Load checkpoint
state = torch.load("best.pt", map_location="cpu", weights_only=False)

# Initialise model (architecture definition required — see GitHub repo)
model = CrossAttentionFusionModel(meta_dim=35)
model.load_state_dict(state["model"])
model.eval()

# Image preprocessing
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

# Metadata: [age_normalised, sex_onehot(3), localization_onehot(31)]
# See GitHub repo for full encode_metadata() function

Training

Trained on Google Colab free tier (T4 GPU) for 30 epochs. Full experimental methodology including architecture ablation, hyperparameter search, and fusion strategy comparison available in the GitHub repository.

Limitations

• Trained on HAM10000 only — performance on other dermoscopy datasets not evaluated
• Patient metadata required at inference time
• Not validated for clinical use

Citation

Tschandl P., Rosendahl C. & Kittler H. The HAM10000 dataset, a large
collection of multi-source dermatoscopic images of common pigmented
skin lesions. Sci. Data 5, 180161 (2018).