dermora-dinov2-vit-b-skin

Dermora — DINOv2 ViT-B Skin Condition Model

A DINOv2 ViT-B model fine-tuned for skin condition classification with stress-correlation awareness. This model is the AI core of Dermora, a psychodermatology-focused health monitoring system that studies the relationship between psychological stress and visible skin conditions.

The model takes a skin image as input and classifies it into condition categories, capturing both the nature of the condition and patterns that may correlate with stress-triggered onset or severity.

---

Model Details

---

Why DINOv2

DINOv2, developed by Meta AI, is a self-supervised vision foundation model trained on approximately 142 million images without labels. It learns highly generalizable visual representations that transfer well across downstream tasks — including medical and dermatological imaging.

Key reasons for choosing DINOv2 as the backbone:

• Strong feature extraction from limited labeled data, which is common in medical datasets
• Patch-based transformer architecture captures fine-grained skin texture and pattern details
• The CLS token provides a compact, expressive image-level representation for classification
• Consistently outperforms CNN-based backbones on transfer learning benchmarks
• Demonstrated effectiveness in dermatology and pathology tasks in prior research

---

Intended Use

Primary Use

• Skin condition classification from images (acne, eczema, psoriasis, normal)
• Stress-correlation inference from visible skin patterns
• Research in psychodermatology and AI-assisted skin health monitoring
• Prototyping and educational use within the Dermora platform

Out-of-Scope Use

• Not intended for autonomous clinical diagnosis
• Not a replacement for dermatologist evaluation
• Not validated for clinical deployment in any setting
• Not suitable for use on data significantly outside the training distribution (e.g., dermoscopy images, histology slides)

---

How to Use

Installation

pip install transformers torch pillow

Basic Inference

from transformers import AutoImageProcessor, AutoModelForImageClassification
from PIL import Image
import torch

# Load model and processor
model = AutoModelForImageClassification.from_pretrained(
    "rachitgoyell/dermora-dinov2-vit-b-skin"
)
processor = AutoImageProcessor.from_pretrained(
    "rachitgoyell/dermora-dinov2-vit-b-skin"
)

# Load and preprocess image
image = Image.open("sample_skin.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

# Run inference
with torch.no_grad():
    outputs = model(**inputs)

logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
predicted_label = model.config.id2label[predicted_class_idx]

print(f"Predicted condition: {predicted_label}")
print(f"Confidence scores: {torch.softmax(logits, dim=-1)}")

Batch Inference

from PIL import Image

images = [
    Image.open("sample1.jpg").convert("RGB"),
    Image.open("sample2.jpg").convert("RGB"),
    Image.open("sample3.jpg").convert("RGB"),
]

inputs = processor(images=images, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

predicted_classes = outputs.logits.argmax(-1).tolist()
for idx, pred in enumerate(predicted_classes):
    print(f"Image {idx + 1}: {model.config.id2label[pred]}")

Extracting Features (without classification head)

from transformers import AutoModel

feature_extractor = AutoModel.from_pretrained(
    "rachitgoyell/dermora-dinov2-vit-b-skin"
)

with torch.no_grad():
    features = feature_extractor(**inputs)

cls_embedding = features.last_hidden_state[:, 0, :]  # CLS token
print("Feature shape:", cls_embedding.shape)  # (batch, 768)

---

Training Details

Dataset

• rachitgoyell/dermoraSkinStressDataset
• Skin condition photographs across acne, eczema, psoriasis, and normal categories
• Annotated with condition labels and stress-correlation indicators

Training Approach

• Transfer learning from facebook/dinov2-base pretrained weights
• Classification head added on top of the CLS token output
• All layers fine-tuned end-to-end on the skin stress dataset

Preprocessing

• Images resized and center-cropped to match DINOv2 input requirements
• Normalization using ImageNet mean and standard deviation
• Augmentation: horizontal flip, random rotation, color jitter, random resized crop

Training Configuration

---

Evaluation

Metrics reported on the held-out test split of the dermoraSkinStressDataset. External validation is ongoing.

These figures reflect internal evaluation and should not be interpreted as clinically validated performance. Independent external validation has not yet been completed.

---

Limitations

• Not clinically validated — the model has not been evaluated in a clinical setting or reviewed by dermatologists for diagnostic accuracy
• Cannot diagnose disease — outputs are research-grade predictions, not medical diagnoses
• Indirect stress inference — stress is not directly observable from skin images; the model captures visual correlates of stress-associated conditions, not stress itself
• Demographic bias — performance may vary across skin tones, ages, and imaging conditions not well-represented in the training data
• Dataset quality dependency — model quality is bounded by the quality and clinical validity of the training annotations
• Distribution sensitivity — the model may perform poorly on image types outside its training distribution (e.g., dermoscopy, clinical close-ups with polarized lighting)

---

Ethical Considerations

• This model is a research and prototyping tool and must not be used to make autonomous clinical decisions
• Stress inference from skin images is indirect and should never be used as a standalone mental health assessment
• Risk of demographic bias across skin tones must be evaluated before any downstream use
• Any deployment in a health-related context requires human clinical oversight and appropriate ethical review
• The model does not store or process personally identifiable information, but skin images themselves are sensitive biometric data and must be handled accordingly

---

Related Resources

• Dataset: rachitgoyell/dermoraSkinStressDataset
• Base model: facebook/dinov2-base

---

License

Released under the MIT License.

This model is intended for research and educational use only. Clinical deployment requires independent validation, institutional review, and applicable regulatory approvals. The authors make no warranties regarding the clinical accuracy or safety of this model.