sift-siglip2

sift-siglip2 — SigLIP 2 (so400m-patch14-384) for Apple Silicon, Core ML

Re-hosted Core ML conversion of google/siglip2-so400m-patch14-384 for Sift, a local photo library app. Public so the app can download embeddings models without tokens. Nothing personal went through this pipeline — validation uses seeded synthetic gradients plus CC0 stock photos, not real library content.

Source and provenance

• Upstream: google/siglip2-so400m-patch14-384
• Pinned revision: e8e487298228002f3d8a82e0cd5c8ea9c567f57f
• License: Apache-2.0 (inherited from upstream)
• Shipped precision: fp16 weights/compute with softmax, layer_norm, all reduce ops, and pow/sqrt/rsqrt kept in fp32 (ct.transform.FP16ComputePrecision)
• Converter: transformers 4.57.1, torch 2.7.0, coremltools 9.0, macOS 15 target

Artifacts

Both towers output the pooled, projected feature — the same vector the upstream get_image_features / get_text_features APIs return. Length normalisation, if wanted, is the caller's step. The fp32 build that previously sat at this path remains in repo history (~4.54 GB zipped).

Input contracts (the Swift side must match exactly)

Image preprocessing (from the upstream preprocessor_config.json):

1. Convert to RGB.
2. Resize to 384×384 (size: {height: 384, width: 384}), bilinear interpolation (resample: 2 = PIL BILINEAR).
3. Rescale by 1/255 (rescale_factor 0.00392156862745098) → [0,1].
4. Normalize per channel: mean 0.5, std 0.5 (image_mean, image_std).

Text preprocessing (Gemma tokenizer from the upstream snapshot):

1. Tokenize with the checkpoint's tokenizer; max length 64, padding="max_length" (right padding, padding_side=right), truncation=True.
2. return_attention_mask=True is required — the Gemma tokenizer omits the mask by default, and the model needs it.
3. Feed input_ids and attention_mask as int32, shape [1, 64].

Output: 1152-dim fp32 feature vector per tower.

Validation

Deterministic gate: worst cosine vs the upstream torch model across all fixed samples, computed against the UNMODIFIED upstream weights (references are generated before the conversion-time pooling-head swap, so the gate measures the rewrite honestly). Gate: cosine ≥ 0.999 on every sample.

The fp16 straight failure was a synthetic high-contrast IMAGE sample; text samples were ≥ 0.999997 in every mode.

Full per-sample numbers in validation.json:

• 4 synthetic gradient/shape images (deterministic)
• 3 real photos, CC0 scikit-image sample data (astronaut/coffee/chelsea): 0.999628 / 0.999397 / 0.999287
• 4 text prompts

The vision tower's attention-pooling head is reimplemented with explicit matmul/softmax for the conversion (same weights; upstream's nn.MultiheadAttention wrapper does not convert to Core ML). The gate above is what verifies that reimplementation is faithful.

Usage in Sift

• Image input: RGB 384×384 per the contract above.
• Text input: Gemma tokenizer per the contract above.
• Embedding dim: 1152.
• Fetcher verifies the sha256 above before unpacking.