Downloads · 30 days
55
3% of all-time downloads
mrzjy/GenshinImpact-ViT-SO400M-14-SigLIP-384
GenshinImpact-ViT-SO400M-14-SigLIP-384 is a zero-shot image classification model from mrzjy. Use it for the zero-shot image classification task on the model card, and read the license before you ship it in a product. It is set up for open_clip. The card lists the license as apache-2.0.
A simple open-sourced SigLIP model fine-tuned on Genshin Impact's image-text pairs.
Downloads · 30 days
55
3% of all-time downloads
All-time downloads
2K
Public
Repo size
7 GB
Likes
2
Public
Click a slice to open those files.
.bin3.5 GB · 50%
From the Hugging Face model README
A simple open-sourced SigLIP model fine-tuned on Genshin Impact's image-text pairs.
Visit the github for case study and data pair examples.
The model is far from being perfect, but could still offer some better text-image matching performance in some Genshin Impact scenarios.
| Model | Checkpoint Size | Val Loss |
|---|---|---|
| GenshinImpact-CLIP-ViT-B-16-laion2B-s34B-b88K | 0.59 GB | 1.152 |
| GenshinImpact-ViT-SO400M-14-SigLIP-384 | 3.51 GB | 0.362 |
You can use the raw model for tasks like zero-shot image classification and image-text retrieval.
Here is how to use this model to perform zero-shot image classification:
import torch
import torch.nn.functional as F
from PIL import Image
import requests
from open_clip import create_model_from_pretrained, get_tokenizer
def preprocess_text(string):
return "Genshin Impact\n" + string
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# load checkpoint from local path
# model_path = "path/to/open_clip_pytorch_model.bin"
# model_name = "ViT-SO400M-14-SigLIP-384"
# model, preprocess = create_model_from_pretrained(model_name=model_name, pretrained=model_path, device=device)
# tokenizer = get_tokenizer(model_name)
# or load from hub
model, preprocess = create_model_from_pretrained('hf-hub:mrzjy/GenshinImpact-ViT-SO400M-14-SigLIP-384')
tokenizer = get_tokenizer('hf-hub:mrzjy/GenshinImpact-ViT-SO400M-14-SigLIP-384')
# image
image_url = "https://static.wikia.nocookie.net/gensin-impact/images/3/33/Qingce_Village.png"
image = Image.open(requests.get(image_url, stream=True).raw)
image = preprocess(image).unsqueeze(0).to(device)
# text choices
labels = [
"This is an area of Liyue",
"This is an area of Mondstadt",
"This is an area of Sumeru",
"This is Qingce Village"
]
labels = [preprocess_text(l) for l in labels]
text = tokenizer(labels, context_length=model.context_length).to(device)
with torch.autocast(device_type=device.type):
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features = F.normalize(image_features, dim=-1)
image_features = F.normalize(image_features, dim=-1)
text_features = F.normalize(text_features, dim=-1)
text_probs = torch.sigmoid(image_features @ text_features.T * model.logit_scale.exp() + model.logit_bias)
scores = [f"{s:.3f}" for i, s in enumerate(text_probs.tolist()[0])]
print(scores) # [0.016, 0.000, 0.001, 0.233]
SigLIP model further fine-tuned on 17k Genshin Impact English text-image pairs at resolution 384x384.
There're currently 17,428 (train) and 918 (validation) text-image pairs used for model training.
All the images and texts are crawled from Genshin Fandom Wiki and are manually parsed to form text-image pairs.
Image Processing:
Text Processing:
<img> tag or specified web content.Data Distribution:

Validation Loss Curve
