CLIP-ViT-B-32-LivingThings-10M

CLIP-ViT-B-32-LivingThings-10M

A CLIP ViT-B/32 model trained from scratch on the LivingThings-10M subset (animals, plants, fungi) of EntityNet-33M.

Trained only on 10M web images of organisms, found by image search with entities from Wikidata and WordNet instead of a hand-curated biology dataset. It outperforms BioCLIP, which was trained on hand-curated data of similar size (TreeOfLife-10M), on species it has never seen (Rare Species).

Paper · Code · Dataset · Project page

Results

Zero-shot top-1 accuracy (%), all models evaluated with the same protocol from the paper, so numbers can differ slightly from those on other model cards. Retrieval is the average image-text recall@1 over COCO, Flickr30k and XM3600. Rare Species tests 400 species that were removed from our training data. Following the paper, it is only reported for models whose training data is known to exclude them ("–" otherwise).

Usage

import torch
import open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:lmb-freiburg/CLIP-ViT-B-32-LivingThings-10M")
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-32", context_length=32)  # trained with 32 text tokens

image = preprocess(Image.open("image.jpg")).unsqueeze(0)
labels = ["red fox", "gray wolf", "coyote"]
text = tokenizer(labels)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(labels[probs.argmax().item()])

Species names work in English ("red fox") and Latin ("Vulpes vulpes"); the paper reports the better of the two.

Training

The model has seen ~0.2B images at a batch size of 8k. Text labels were sampled 50/50 from the images' web alt texts and from the knowledge-graph text behind the image (the search query, or the entity's aliases or descriptions). See the paper for details.

All EntityNet models

Citation

@inproceedings{ging2026entitynet,
  author    = {Simon Ging and Sebastian Walter and Jelena Bratuli{\'c} and Johannes Dienert and Hannah Bast and Thomas Brox},
  title     = {Using Knowledge Graphs to Harvest Datasets for Efficient {CLIP} Model Training},
  booktitle = {Pattern Recognition, 47th {DAGM} German Conference, {DAGM} {GCPR} 2025, Freiburg, Germany, September 23--26, 2025, Proceedings},
  series    = {Lecture Notes in Computer Science},
  publisher = {Springer Nature Switzerland},
  year      = {2026},
  pages     = {287--302},
  isbn      = {978-3-032-12840-9},
  doi       = {10.1007/978-3-032-12840-9_19}
}