ViT-H-14-laion2b_s32b_b79k

Fine-Tuned OpenCLIP ViT-H-14 Checkpoints (LAION-2B)

This repository contains full fine-tuned OpenCLIP ViT-H-14 checkpoints, each fine-tuned independently on one of the eight Vision8 downstream vision datasets starting from laion2b_s32b_b79k.

Contents

Each dataset folder contains:

• full_best_ep.pt: checkpoint with the best validation accuracy
• full_last_ep.pt: checkpoint from the final training epoch

Datasets and epochs:

Vision8 macro-average: 94.79%.

Training configuration

• Model: OpenCLIP ViT-H-14
• Pretrained weights: laion2b_s32b_b79k
• Full checkpoint state dict, fp32
• AdamW, learning rate 1e-5, weight decay 0.1
• Effective batch size 128 (batch_size=4, gradient accumulation 32)
• Cosine scheduler
• Gradient clipping: 1.0
• Early stopping disabled
• Validation fraction: 0.1
• Seed: 42

Usage

The checkpoint is a metadata-wrapped payload. Its state_dict uses the project prefix clip_model.model.; remove that prefix before loading it into the OpenCLIP model:

import torch
import open_clip
from huggingface_hub import hf_hub_download

checkpoint = hf_hub_download(
    repo_id="fillo-rinaldi/ViT-H-14-laion2b_s32b_b79k",
    filename="Cars/full_best_ep.pt",
    repo_type="model",
)
payload = torch.load(checkpoint, map_location="cpu", weights_only=False)
state_dict = {
    key.removeprefix("clip_model.model."): value
    for key, value in payload["state_dict"].items()
    if key.startswith("clip_model.model.")
}

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-H-14",
    pretrained=None,
)
model.load_state_dict(state_dict, strict=True)

The .pt files contain format="full", OpenCLIP backbone metadata, and the complete OpenCLIP model state dict.