tipsv2-so400m14

See our collection for all versions of TIPSv2.

Run TIPSv2 with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/tipsv2-so400m14

Paper: TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment (arXiv:2604.12012)

TIPSv2 (Google DeepMind) is a CLIP/SigLIP-style dual encoder: a DINOv2-style ViT vision tower with register tokens plus a bidirectional text tower, aligned with a temperature-scaled contrastive objective.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of google/tipsv2-so400m14 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX. The full model and both towers load from this single repo.

✨ Quick start (zero-shot)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
import numpy as np
import keras
from zeromodels.models.tipsv2 import Tipsv2Model, Tipsv2Processor

model = Tipsv2Model.from_weights("zeromodels/tipsv2-so400m14")
processor = Tipsv2Processor.from_weights("zeromodels/tipsv2-so400m14")

image = Image.open("your_image.jpg").convert("RGB")
texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=texts, images=np.array(image))
out = model(inputs)
probs = keras.ops.softmax(out["logits_per_image"], axis=-1)
print(keras.ops.convert_to_numpy(probs)[0])

Towers only:

from zeromodels.models.tipsv2 import Tipsv2VisionModel, Tipsv2TextModel
vision = Tipsv2VisionModel.from_weights("zeromodels/tipsv2-so400m14")
text = Tipsv2TextModel.from_weights("zeromodels/tipsv2-so400m14")

All TIPSv2 variants load the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• The image processor rescales to [0, 1] (no mean/std normalization); input resolution is 448.
• Upstream checkpoints: Tipsv2Model.from_weights("hf:google/tipsv2-so400m14").

Special Thanks

A huge thank you to the TIPSv2 authors (Google DeepMind) and the HF community.

License: Apache-2.0 (matches the upstream google/tipsv2-so400m14 checkpoint).