CLIP-ViT-B-16-EntityNet-33M

CLIP-ViT-B-16-EntityNet-33M

A CLIP ViT-B/16 model trained from scratch on EntityNet-33M.

EntityNet is small enough to train a CLIP model from scratch on a single 8-GPU machine in about two days, which makes it useful for research that needs full control over the training data. It is especially strong on animals and plants.

Paper · Code · Dataset · Project page

Results

Zero-shot top-1 accuracy (%), all models evaluated with the same protocol from the paper, so numbers can differ slightly from those on other model cards. Retrieval is the average image-text recall@1 over COCO, Flickr30k and XM3600. Rare Species tests 400 species that were removed from our training data. Following the paper, it is only reported for models whose training data is known to exclude them ("–" otherwise).

Usage

import torch
import open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:lmb-freiburg/CLIP-ViT-B-16-EntityNet-33M")
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-16", context_length=32)  # trained with 32 text tokens

image = preprocess(Image.open("image.jpg")).unsqueeze(0)
labels = ["a dog", "a cat", "a rabbit"]
text = tokenizer(labels)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(labels[probs.argmax().item()])

Species names work in English ("red fox") and Latin ("Vulpes vulpes"); the paper reports the better of the two.

Training

The model has seen ~0.6B images at a batch size of 8k. Text labels were sampled 50/50 from the images' web alt texts and from the knowledge-graph text behind the image (the search query, or the entity's aliases or descriptions). See the paper for details.

All EntityNet models

Citation

@inproceedings{ging2026entitynet,
  author    = {Simon Ging and Sebastian Walter and Jelena Bratuli{\'c} and Johannes Dienert and Hannah Bast and Thomas Brox},
  title     = {Using Knowledge Graphs to Harvest Datasets for Efficient {CLIP} Model Training},
  booktitle = {Pattern Recognition, 47th {DAGM} German Conference, {DAGM} {GCPR} 2025, Freiburg, Germany, September 23--26, 2025, Proceedings},
  series    = {Lecture Notes in Computer Science},
  publisher = {Springer Nature Switzerland},
  year      = {2026},
  pages     = {287--302},
  isbn      = {978-3-032-12840-9},
  doi       = {10.1007/978-3-032-12840-9_19}
}