medical-models

Medical Imaging Models

This repository contains two complementary medical imaging AI model collections:

1. BiomedCLIP Router - Routes medical images to the correct imaging modality
2. Disease-Specific Models - Classification and segmentation models for various medical imaging domains

---

Model 1: BiomedCLIP Router

Description

A modality classification model that routes medical images to one of 7 imaging categories. Built on Microsoft's BiomedCLIP backbone, this router enables automatic sorting of medical images to the appropriate downstream model.

Supported Modalities

Performance

Training Details

• Base Model: microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
• Training Images: 14,000 (2,000 per class)
• Validation Images: 1,750 (250 per class)
• Test Images: 1,750 (250 per class)
• Best Epoch: 1
• Validation Macro F1: 99.94%

Files

biomedclip_router_20260423_194004/
├── best_biomedclip_router.pt    # Trained model weights (748MB)
├── metrics.json                  # Training metrics and history
├── train.log                     # Training logs
├── session_name.txt              # Session identifier
└── unseen_holdout_300/          # Holdout test data info

---

Model 2: Disease-Specific Models

A collection of specialized models for disease classification and segmentation across 8 medical imaging domains.

---

2.1 Breast Ultrasound

Classification Model

Task: Classify breast ultrasound images as benign, malignant, or normal

Classes: benign, malignant, normal

Segmentation Model

Task: Segment lesions in breast ultrasound images

Files:

breast_ultrasound/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

---

2.2 Chest X-Ray

Classification Model

Task: 4-class chest X-ray classification (COVID-19 Radiography Database)

Classes: covid, lung_opacity, normal, viral_pneumonia

• Architecture: DenseNet121 (torchvision IMAGENET1K_V1, the CheXNet chest-X-ray backbone) + fresh 4-class head (replaces the previous 15-class BiomedCLIP multi-label model that scored ~21% exact-match accuracy)
• Dataset: COVID-19 Radiography Database (Kaggle tawsifurrahman/covid19-radiography-database); also mirrored at dataset repo umairinayat/fyp-dataset
• Deduplication: near-duplicates removed via MD5 + perceptual pHash (mirror-aware) -> 19,133 unique images
• Split: stratified 70/15/15 (train 13,393 / val 2,870 / test 2,870), deduplicated globally before splitting (no train/test leakage)
• Input: 224x224, ImageNet normalization
• Training: 15 epochs, AdamW + OneCycleLR (max_lr 2e-4, weight_decay 0.03), tempered class-weighted CrossEntropy (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing (train only); no test-time augmentation

Images per class (after dedup; total unique = 19,133):

Segmentation Model

Task: Binary lung-region segmentation (using the dataset's masks/)

• Loss: BCEWithLogits + Dice; same leakage-free split as classification; image & mask resized to 256x256; train augmentation = flip + rot90

Files:

chest_xray/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   ├── training_history.json
│   └── plots/
└── segmentation/
    ├── best_segmenter.pt
    ├── metrics.json
    ├── training_history.json
    └── plots/

---

2.3 Dermatology

Classification Model

Task: Classify skin lesions as malignant or non-malignant (2-class)

Classes: malignant, non_malignant

• Architecture: ConvNeXt-Small (torchvision IMAGENET1K_V1) + 2-class head (replaces the previous BiomedCLIP model at 88.74%)
• Dataset: combined 2 sources for variety - HAM10000 (Kaggle surajghuwalewala/ham1000-segmentation-and-classification, 7->2 class) + fanconic/skin-cancer-malignant-vs-benign
• Leakage-free split: HAM10000 has multiple images per lesion, so it is split by lesion_id (all images of a lesion stay in one split); fanconic uses its own official train/test holdout -> train 9,667 / val 1,489 / test 2,156
• Training: 12 epochs @ 224px, AdamW + OneCycleLR, tempered class-weighted CE (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing

Images per class:

Segmentation Model

Task: Binary lesion segmentation (HAM10000 lesion masks)

• Architecture: SegFormer-B3 (nvidia/segformer-b3-finetuned-ade-512-512, ADE20K-pretrained) with a 1-label binary head
• Dataset: HAM10000 lesion masks (Kaggle surajghuwalewala/ham1000-segmentation-and-classification), 10,015 image+mask pairs
• Split: lesion-level (by lesion_id) so the same lesion never appears in both train and test -> train 6,987 / val 1,478 / test 1,550 (no leakage)
• Training: 18 epochs @ 256px, AdamW + OneCycleLR (max_lr 6e-5), BCEWithLogits + Dice, mixed precision; augmentation = hflip + rot90

Files:

dermatology/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmenter.pt
    ├── metrics.json
    └── plots/

---

2.4 Endoscopy

Classification Model

Task: Classify GI endoscopy images into 5 classes (healthy added vs. previous 4-class model)

Classes: barretts, esophagitis, polyp, ulcerative_colitis, healthy

• Architecture: ConvNeXt-Small (torchvision IMAGENET1K_V1) + fresh 5-class head (replaces the previous 4-class BiomedCLIP model)
• Dataset: HyperKvasir labeled images (23 source classes mapped to 5 targets); barretts<-{barretts,barretts-short-segment}, esophagitis<-{esophagitis-a,esophagitis-b-d}, polyp<-{polyps}, ulcerative_colitis<-{ulcerative-colitis-grade-*}, healthy<-{cecum,pylorus,z-line,retroflex-rectum,retroflex-stomach,ileum}
• Deduplication: near-duplicates removed via MD5 + perceptual pHash (mirror-aware) -> 6,695 unique images
• Split: stratified 70/15/15 (train 4,686 / val 1,004 / test 1,005), deduplicated globally before splitting (no train/test leakage)
• Input: 384x384, ImageNet normalization
• Training: 22 epochs, AdamW + OneCycleLR (max_lr 2e-4, weight_decay 0.05), tempered class-weighted CrossEntropy (label smoothing 0.1), mixed precision; augmentation = RandomResizedCrop + TrivialAugmentWide + flip + RandomErasing (train only)
• Eval: no test-time augmentation (single center-crop)
• Note: barretts has only 94 images in all of HyperKvasir (66 train / 14 test), so it remains the hardest class and caps macro-F1; the other 4 classes score F1 0.84-0.99

Images per class (after dedup; total unique = 6,695):

Segmentation Model

Task: Segment regions of interest in endoscopy images

Files:

endoscopy/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

---

2.5 Mammography

Classification Model

Task: Classify mammogram patches as benign, malignant, or normal (3-class)

Classes: benign, malignant, normal

• Architecture: ResNet50 (torchvision IMAGENET1K_V2 pretrained weights) + fresh 3-class linear head
• Dataset: INbreast + MIAS + DDSM (CLAHE preprocessed) - 26,602 raw images
• Deduplication: 10,472 duplicate / near-duplicate images removed via MD5 + perceptual pHash (LSH banding, mirror-aware, Hamming <= 5) -> 16,130 unique images
• Split: stratified 70/15/15 (train 11,291 / val 2,419 / test 2,420), deduplicated globally before splitting, so the held-out test set contains no image (or near-duplicate) seen in train/val - no leakage
• Input: 224x224, grayscale replicated to 3 channels, ImageNet normalization
• Training: 15 epochs, AdamW + OneCycleLR (max_lr 3e-4), class-weighted CrossEntropy, mixed precision
• Augmentation (train only): RandomResizedCrop, RandomHorizontalFlip, RandomRotation(15), RandomAffine, ColorJitter, RandomErasing

Images per class (after dedup; total unique = 16,130):

Replaces the previous 2-class (BENIGN/MALIGNANT) BiomedCLIP mammography classifier (was 66% accuracy). The new model is a ResNet50 - load it with torchvision.models.resnet50 (see Usage below), not the BiomedCLIP loader.

Test confusion matrix (rows = true, cols = predicted benign, malignant, normal):

benign:    [963,    3,    0]
malignant: [  6, 1157,    0]
normal:    [  0,    0,  291]

Segmentation Model

Task: Binary mass segmentation in mammography (CBIS-DDSM)

• Architecture: SegFormer-B3 (nvidia/segformer-b3-finetuned-ade-512-512, ADE20K-pretrained), fine-tuned with a 1-label binary head — replaces the previous UNet++ (86.31% Dice)
• Dataset: CBIS-DDSM mass cases (Kaggle awsaf49/cbis-ddsm-breast-cancer-image-dataset); full mammogram + full-image ROI mask paired, square patch cropped around each lesion (2x margin) -> 512x512
• Split: official CBIS-DDSM mass train/test (patient-disjoint, no leakage); 15% of train held as validation -> train 979 / val 172 / test 341 patches
• Training: 15 epochs @ 384px, AdamW + OneCycleLR (max_lr 6e-5), BCEWithLogits + Dice, mixed precision; augmentation = hflip/vflip/rot90
• Eval: no test-time augmentation

Files:

mammography/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

---

2.6 Thyroid Ultrasound

Classification Model

Task: Classify thyroid nodules by risk level

Classes: low_risk, suspicious

Segmentation Model

Task: Segment thyroid nodules in ultrasound images

Files:

thyroid_ultrasound/
├── classification/
│   ├── best_classifier.pt
│   ├── metrics.json
│   └── plots/
└── segmentation/
    ├── best_segmentor.pt
    ├── metrics.json
    └── plots/

---

Performance Summary

Classification Models Performance

Mammography uses ResNet50, Endoscopy uses ConvNeXt-Small, and Chest X-Ray uses DenseNet121 (other disease classifiers use BiomedCLIP ViT-B/16).

Segmentation Models Performance

---

Dataset Statistics (per-class splits)

Counts = train / val / test images after de-duplication. "Unique total" = images actually used. Legacy BiomedCLIP / SMP models (Breast US, Thyroid US, Endoscopy seg, Fundus, Microscopy) were trained externally before this work; their per-class splits were not recorded and are marked N/R.

BiomedCLIP Router (7 modalities)

Mammography — Classification (3-class) · INbreast+MIAS+DDSM (CLAHE), deduped

Mammography — Segmentation · CBIS-DDSM mass (binary patches, official patient split)

Chest X-Ray — Classification (4-class) · COVID-19 Radiography, deduped

Chest X-Ray — Segmentation · COVID-19 (binary lung, same images)

Endoscopy — Classification (5-class, healthy added) · HyperKvasir, deduped

Dermatology — Classification (2-class) · HAM10000 (lesion-level) + fanconic

Dermatology — Segmentation · HAM10000 lesion masks (binary, lesion-level split)

Legacy models (per-class splits not recorded)

---

Usage

Installation

pip install torch torchvision huggingface_hub open_clip_torch segmentation-models-pytorch

Download Models

# Download entire repository
hf download umairinayat/medical-models --local-dir ./medical-models

# Download specific components
hf download umairinayat/medical-models biomedclip_router_20260423_194004 --local-dir ./router
hf download umairinayat/medical-models disease_models --local-dir ./disease_models

Router Inference Example

import torch
import open_clip
from PIL import Image

# Load BiomedCLIP Router
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
    'hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224'
)
tokenizer = open_clip.get_tokenizer(
    'hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224'
)

# Load trained router weights
checkpoint = torch.load('biomedclip_router_20260423_194004/best_biomedclip_router.pt', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])

# Inference
image = preprocess_val(Image.open('medical_image.png')).unsqueeze(0)
with torch.no_grad():
    output = model(image)
    predicted_class = output.argmax(dim=1).item()

class_names = ['Endoscopy', 'Dermatology', 'X-Ray', 'Ultrasound', 
               'Mammography', 'Fundus/Retinography', 'Microscopy']
print(f"Predicted modality: {class_names[predicted_class]}")

Mammography Classification Inference (ResNet50)

import torch
from PIL import Image
from torchvision import models, transforms
from huggingface_hub import hf_hub_download

# Load checkpoint
ckpt_path = hf_hub_download(
    'umairinayat/medical-models',
    'disease_models/mammography/classification/best_classifier.pt')
ckpt = torch.load(ckpt_path, map_location='cpu', weights_only=False)
class_names = ckpt['class_names']  # ['benign', 'malignant', 'normal']

# Build model
model = models.resnet50()
model.fc = torch.nn.Linear(model.fc.in_features, len(class_names))
model.load_state_dict(ckpt['model_state'])
model.eval()

# Preprocess (224x224, grayscale -> RGB, ImageNet stats)
tfm = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(ckpt['normalization_mean'], ckpt['normalization_std']),
])
img = Image.open('mammogram.png').convert('RGB')
x = tfm(img).unsqueeze(0)
with torch.no_grad():
    probs = torch.softmax(model(x), dim=1)[0]
print({c: round(float(p), 4) for c, p in zip(class_names, probs)})

---

Model Architecture

BiomedCLIP Router

• Backbone: ViT-B/16 (Vision Transformer)
• Pre-training: BiomedCLIP (PubMed 15M+ biomedical articles)
• Classification Head: Linear layer with 7 output classes
• Input Size: 224 x 224 pixels

Classification Models

• Backbone: BiomedCLIP ViT-B/16 (except Mammography, which uses ResNet50, and Endoscopy, which uses ConvNeXt-Small)
• Fine-tuned: Full model (backbone unfrozen)
• Augmentation: Test-time augmentation (TTA) enabled

Segmentation Models

• Architecture: SMP UNet++ (UNet with nested skip pathways)
• Encoder: Pretrained on ImageNet
• Loss: Weighted Dice + BCE Loss
• Augmentation: Test-time augmentation (TTA) enabled

---

Intended Use

These models are designed for:

• Medical image triage and routing
• Automated modality classification
• Disease detection and diagnosis support
• Medical image segmentation
• Medical imaging research

---

Limitations

• Models trained on specific datasets; performance may vary on different populations
• Chest X-Ray model shows lower performance due to multi-label complexity and class imbalance
• Not a replacement for professional medical diagnosis
• Should be validated before clinical deployment
• May not generalize well to out-of-distribution images

---

Training Configuration

All classification models were trained with:

• Base Model: hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
• Backbone: Unfrozen (fine-tuned)
• Test-Time Augmentation: Enabled

Exception - Mammography: trained with torchvision.resnet50 (IMAGENET1K_V2), a fresh 3-class head, AdamW + OneCycleLR, class-weighted CrossEntropy, on a deduplicated INbreast+MIAS+DDSM (CLAHE) split with leakage-free train/val/test separation. See disease_models/mammography/classification/metrics.json.

All segmentation models were trained with:

• Architecture: SMP UNet++
• Loss: Combined Dice + BCE with positive class weighting
• Test-Time Augmentation: Enabled

---

Citation

If you use these models, please cite:

@misc{medical-models-2026,
  author = {Umair Inayat},
  title = {Medical Imaging Models: BiomedCLIP Router and Disease-Specific Classifiers},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/umairinayat/medical-models}
}

---

License

MIT License

---

Contact

For questions or issues, please open an issue on the Hugging Face repository.

---

Models Trained in This Work

Models trained and pushed during the current update cycle (each under disease_models/<disease>/{classification,segmentation}/, with weights + metrics.json + training_history.json + plots).

Dataset also pushed: umairinayat/fyp-dataset — COVID-19 Radiography (21,165 images + masks).

Trained but not pushed (came out below the existing repo models, so the originals were kept):

• Breast Ultrasound classification (74.5%) and segmentation (70% Dice) — BUSI is small (~626 unique images), honest held-out results were below the existing 92.3% / 83.8%.
• Endoscopy segmentation — SegFormer-B3 (88.26% Dice) and UNet (86%) were at/below the existing 88.5% on a noisy 150-image test.

Untouched (original/legacy models retained): BiomedCLIP Router, Breast Ultrasound, Thyroid Ultrasound, Fundus Retinography, Microscopy, and Endoscopy segmentation.