jina-clip-v2-split-onnx

Jina CLIP v2 split ONNX

Independent text and vision ONNX encoders derived from jinaai/jina-clip-v2.

Every model is self-contained (FP32 uses an external-data companion file), supports dynamic batch sizes, and returns normalized float32 embeddings named embeddings with shape [batch, 1024]. The text models also support dynamic sequence lengths.

Which runtime should I use?

Measured on RTX 2070 (sm_75) unless noted. Full methodology: GitHub results.md.

Notes:

• TensorRT fuses each attention layer into a single flash-attention-style kernel (_gemm_mha_v2) with linear memory; engines are hardware-specific and bound to the built optimization profile (default [1,8]..[32,2048]; pad shorter sequences with pad-token id 1, which the internal mask makes semantically neutral). GTX 1080 Ti / Pascal cannot run TensorRT 10+ — use ONNX Runtime there.
• FP16 ONNX is not recommended for CPU inference: CPU providers promote unsupported FP16 ops or insert casts, increasing latency and memory.

Preprocessing and inference

Use the tokenizer and image processor from the base model:

import onnxruntime as ort
from transformers import AutoImageProcessor, AutoTokenizer

base_model = "jinaai/jina-clip-v2"
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
image_processor = AutoImageProcessor.from_pretrained(base_model, trust_remote_code=True)

text_session = ort.InferenceSession("jina-clip-v2-text-int8.onnx")
tokens = tokenizer(["A photo of a cat"], padding=True, truncation=True, return_tensors="np")
text_embeddings = text_session.run(["embeddings"], {"input_ids": tokens.input_ids})[0]

# `images` is a list of PIL images.
vision_session = ort.InferenceSession("jina-clip-v2-vision-int8.onnx")
pixels = image_processor(images=images, return_tensors="np").pixel_values
image_embeddings = vision_session.run(["embeddings"], {"pixel_values": pixels})[0]

Matryoshka dimensions

The trained dimensions are 32, 64, 128, 256, 512, 768, and 1024. To reproduce the base model's truncate_dim=N behavior, truncate the prefix and normalize it again:

import numpy as np

embedding = embedding[:, :dimension]
embedding /= np.linalg.norm(embedding, axis=1, keepdims=True)

Validation

Both INT8 graphs were re-quantized on 2026-09-01 with per-channel weight scales and ORT's reduce_range; the artifacts published before that date were per-tensor and are not faithful — a single caption embedded at mean cosine 0.70 to the base model. If you cached the earlier files, re-download them.

The text encoder is scored on 424 real captions against ground-truth PyTorch fp32, on two metrics, because neither catches the other's failure mode: per-vector cosine is absolute and per-document, retrieval ranking is relative and cross-document, and this export has shipped a build that passed each while failing the other.

The FP16 and FP32 graphs score 1.0000 on the same check. Ranking is a 20-profile retrieval fixture in which the mean-pooled vectors of 4 relevant profiles must outrank 16 irrelevant ones; dim=32 ranks wrongly for every build including FP16 and is a property of that fixture, not of any encoder. Reproduce with jina-clip-verify --text-model <graph> from the GitHub repository.

Vision INT8 scores mean cosine 0.9874 (worst 0.9831) at 512 dimensions and 0.9855 (worst 0.9809) at 1024 against the FP16 vision graph on synthetic structured images; a real-image vision gate is still open.

INT8 CPU performance

ONNX Runtime 1.29, Intel i9-7980XE, ORT_ENABLE_BASIC, mean of 5 runs:

Peak resident memory: 2.6 GiB INT8 vs 5.0 GiB FP16. The text input remains dynamic up to 8192 tokens, although ordinary full attention still has quadratic memory use; the TensorRT path removes that quadratic term on supported GPUs.

License and attribution

This is a quantized and structurally split derivative of jinaai/jina-clip-v2. It retains the base model's CC BY-NC 4.0 license and non-commercial restriction. Refer to the base model card for intended use, limitations, training details, and complete attribution.