Downloads · 30 days
20
29% of all-time downloads
rwightman/naflex_ViT-B-32.cc12m-pj24-32-40-siglip
naflex_ViT-B-32.cc12m-pj24-32-40-siglip is a zero-shot image classification model from rwightman. Use it for the zero-shot image classification task on the model card, and read the license before you ship it in a product. It is set up for open_clip. The card lists the license as apache-2.0.
Two-tower CLIP with a NaFlexVit B/32 vision tower, trained on CC12M with the SigLIP sigmoid pairwise loss and patch-size jitter at 24 / 32 / 40 px. The text tower uses classic CLIP BPE (49408 tokens, context length 77…
Downloads · 30 days
20
29% of all-time downloads
All-time downloads
70
Public
Repo size
1.2 GB
Likes
0
Public
Click a slice to open those files.
.bin605 MB · 50%
From the Hugging Face model README
Two-tower CLIP with a NaFlexVit B/32 vision tower, trained on CC12M with the SigLIP sigmoid pairwise loss and patch-size jitter at 24 / 32 / 40 px. The text tower uses classic CLIP BPE (49408 tokens, context length 77). The exported config enables patch-embedding interpolation and includes the learned logit bias. Best measured ImageNet-1k zero-shot top-1 is 43.778% at patch 32 / 100 image tokens; see the full comparison with the InfoNCE patch-jitter run below.
Research checkpoint from a NaFlex patch-size jitter experiment on CC12M. The comparison below covers these two trained runs; it does not estimate variation across seeds.
--siglip --siglip-chunk-size 1024 --loss-dist-impl gathernaflex_patch_interp=trueNaFlex inference at patch 32 / 100 tokens, the best setting in the grid below. This example uses the same evaluation transform as the sweep and requires timm with NaFlex support (evaluated with timm 1.0.28 and PyTorch 2.9.1).
import torch
import open_clip
from PIL import Image
model_id = 'hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40-siglip'
model, _, eval_transform = open_clip.create_model_and_transforms(
model_id, aug_cfg={'use_timm': True, 'naflex': True},
)
model.eval()
tokenizer = open_clip.get_tokenizer(model_id)
patch_size, token_budget = 32, 100
preprocess = eval_transform(
max_seq_len=token_budget,
patch_size=(patch_size, patch_size),
flatten_patches=patch_size == 32, # preserve spatial dimensions for non-base patches
)
sample = preprocess(Image.open('image.jpg').convert('RGB'))
image = {key: value.unsqueeze(0) for key, value in sample.items()}
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.inference_mode():
image_features = model.encode_image(image, normalize=True)
text_features = model.encode_text(text, normalize=True)
logits = model.logit_scale.exp() * image_features @ text_features.T + model.logit_bias
scores = logits.sigmoid()
Change patch_size and token_budget to select another grid cell. The example processes a single
image; batches of different aspect ratios need NaFlex collation and validity masks. The sigmoid
scores are independent image-text match scores. The zero-shot grid uses the 80-prompt class ensemble.
Zero-shot ImageNet-1k top-1 (%) over evaluation patch size x image token budget, comparing this SigLIP checkpoint with the InfoNCE patch-jitter checkpoint. Both are epoch-32 CC12M runs trained with patch sizes 24 / 32 / 40 and sequence lengths 36 / 49 / 64 / 81, using NaDAMuon at lr 3e-3, 2000 warmup steps, and 200704 image tokens per local batch. Token budgets 100-256 and patch sizes 16 / 48 extend beyond the training choices.
Both grids were evaluated together with scripts/naflex_imagenet_sweep.py (revision 9782671):
all 50000 ImageNet-1k validation images, the OpenAI 80-prompt ensemble, BF16 autocast, batch size 256,
the trainer's NaFlex evaluation transform, and patch-embedding interpolation enabled for both models.
Images retain their aspect ratio and are resized to fit the patch budget. For a square image, the
approximate resolution is patch_size * sqrt(token_budget); patch 32 / 100 tokens is about 320 px.
This model: SigLIP top-1 (%)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | 31.182 | 32.970 | 34.174 | 34.818 | 35.340 | 35.064 | 34.238 | 33.112 |
| 24 | 40.284 | 41.642 | 42.600 | 43.102 | 43.636 | 43.666 | 43.162 | 42.676 |
| 32 | 40.492 | 41.778 | 42.698 | 43.226 | 43.778 | 43.674 | 43.220 | 42.792 |
| 40 | 40.210 | 41.506 | 42.568 | 42.966 | 43.576 | 43.592 | 43.126 | 42.744 |
| 48 | 39.962 | 41.404 | 42.456 | 42.762 | 43.498 | 43.588 | 43.130 | 42.768 |
InfoNCE patch jitter: rerun top-1 (%)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | 29.116 | 31.008 | 32.338 | 33.234 | 33.758 | 33.618 | 32.920 | 31.494 |
| 24 | 38.080 | 39.624 | 40.364 | 41.084 | 41.220 | 41.218 | 40.572 | 39.692 |
| 32 | 38.290 | 39.720 | 40.388 | 41.164 | 41.400 | 41.354 | 40.658 | 39.732 |
| 40 | 38.156 | 39.478 | 40.252 | 40.994 | 41.010 | 41.118 | 40.502 | 39.642 |
| 48 | 37.988 | 39.238 | 40.190 | 40.866 | 41.002 | 41.146 | 40.532 | 39.676 |
SigLIP minus InfoNCE (percentage points)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | +2.066 | +1.962 | +1.836 | +1.584 | +1.582 | +1.446 | +1.318 | +1.618 |
| 24 | +2.204 | +2.018 | +2.236 | +2.018 | +2.416 | +2.448 | +2.590 | +2.984 |
| 32 | +2.202 | +2.058 | +2.310 | +2.062 | +2.378 | +2.320 | +2.562 | +3.060 |
| 40 | +2.054 | +2.028 | +2.316 | +1.972 | +2.566 | +2.474 | +2.624 | +3.102 |
| 48 | +1.974 | +2.166 | +2.266 | +1.896 | +2.496 | +2.442 | +2.598 | +3.092 |
This checkpoint has only been evaluated on ImageNet-1k so far; no ImageNet robustness, COCO retrieval, or captioning results are reported here. These are comparisons of two runs, without a multi-seed estimate of training variance.
For the open_clip trainer, use --use-naflex --naflex-patch-sizes 32 --naflex-seq-lens 100 for the
best measured setting. When loading the original training checkpoint by its base model name, also
pass --siglip --force-naflex-patch-interp; the Hub config already preserves the bias and interpolator.