mobileclip2-s0-onnx

<div align="center"> <h1 align="center">MOBILECLIP2-S0 (ONNX & INT8 Quantized)</h1>

Lightweight, CPU-native vision-language embedding model based on Apple's MobileCLIP2 architecture.
Optimized with ONNX Runtime and INT8 dynamic quantization for efficient, zero-GPU inference on laptops, mobile devices, and edge systems.

</div>

---

Model Overview

mobileclip2-s0 provides standalone ONNX weights and INT8 dynamic quantized variants of Apple's MobileCLIP2-S0 architecture, optimized for zero-GPU inference with onnxruntime across Python, C++, Rust, and mobile/WASM runtimes.

• Architecture: FastViT hybrid vision backbone (approx. 12M params) + Transformer text encoder (approx. 15M params)
• Image Input: 256 x 256 RGB (shortest-edge resize + center crop, ImageNet normalized)
• Embedding Output: 512 dimensions (L2-normalized unit vectors)
• Context Length: 77 tokens
• Total Parameters: approx. 27M

---

Deployment Profiles & Benchmarks

Choose the configuration that best matches your target hardware constraints:

• Text Transformer: Quantizes with dynamic INT8, cutting model size by 75% while accelerating CPU text encoding by 2.09x with high fidelity (0.9556 cosine similarity).
• FastViT Vision: Dynamic INT8 on CPU introduces conversion overhead on depthwise convolutions. Thus, FP32 vision + INT8 text (Hybrid: 105 MB) delivers the best balance of speed (122 ms total latency), accuracy, and memory footprint.
• Numerical Parity: FP32 ONNX outputs maintain cosine similarity >= 0.9998 against original PyTorch weights.
• Frame Embedding Latency: 25.0 ms per frame on CPU.

---

Available Model Artifacts

---

Quickstart

1. Download Files

Download only the files you need using huggingface_hub:

from huggingface_hub import hf_hub_download

# Download recommended Hybrid configuration (105 MB total)
vision_path = hf_hub_download(repo_id="felixhrdyn/mobileclip2-s0-onnx", filename="vision_model.onnx")
text_path = hf_hub_download(repo_id="felixhrdyn/mobileclip2-s0-onnx", filename="text_model_quantized.onnx")
tokenizer_path = hf_hub_download(repo_id="felixhrdyn/mobileclip2-s0-onnx", filename="tokenizer.json")

2. Standalone Inference with ONNX Runtime

pip install onnxruntime numpy tokenizers pillow huggingface-hub

import numpy as np
import onnxruntime as ort
from PIL import Image
from tokenizers import Tokenizer

# 1. Load ONNX sessions (Hybrid configuration)
vis_sess = ort.InferenceSession("vision_model.onnx", providers=["CPUExecutionProvider"])
txt_sess = ort.InferenceSession("text_model_quantized.onnx", providers=["CPUExecutionProvider"])
tokenizer = Tokenizer.from_file("tokenizer.json")

# 2. Preprocess & encode image
def preprocess_image(image_path: str) -> np.ndarray:
    img = Image.open(image_path).convert("RGB")
    w, h = img.size
    scale = 256.0 / min(w, h)
    img = img.resize((int(w * scale), int(h * scale)), Image.Resampling.BILINEAR)
    w, h = img.size
    img = img.crop(((w - 256) // 2, (h - 256) // 2, (w - 256) // 2 + 256, (h - 256) // 2 + 256))
    
    arr = (np.array(img, dtype=np.float32) / 255.0 - [0.48145466, 0.4578275, 0.40821073]) / [0.26862954, 0.26130258, 0.27577711]
    return arr.transpose(2, 0, 1)[np.newaxis, ...].astype(np.float32)

pixel_values = preprocess_image("sample.jpg")
input_name = vis_sess.get_inputs()[0].name
img_emb = vis_sess.run(None, {input_name: pixel_values})[0]
img_emb /= np.linalg.norm(img_emb, axis=-1, keepdims=True)

# 3. Preprocess & encode text
encoded = tokenizer.encode("a photo of a golden retriever playing in grass")
tokens = np.array([encoded.ids[:77] + [0] * (77 - len(encoded.ids[:77]))], dtype=np.int64)
txt_input_name = txt_sess.get_inputs()[0].name
txt_emb = txt_sess.run(None, {txt_input_name: tokens})[0]
txt_emb /= np.linalg.norm(txt_emb, axis=-1, keepdims=True)

# 4. Compute cosine similarity
similarity = float(np.dot(img_emb[0], txt_emb[0]))
print(f"Cosine Similarity: {similarity:.4f}")

---

License & Attribution

• Original model architecture and weights by Apple Inc. under Apple Sample Code / MIT license.
• ONNX conversion and quantization maintained by @felixhrdyn.
• Used in applications such as Amon Hen for CPU-native semantic search.