openvision2-vit-huge-patch14-224-vision-only

OpenVision2 · ViT-H/14 @224 — vision encoder + caption decoder

This repo now contains the full OpenVision2 generative model at ViT-H/14, 224px: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image → a descriptive caption.

The encoder files are unchanged; only the decoder (and this card) were added.

Files

Decoder architecture

A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder): the ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and text is generated causally while attending to all image tokens.

24 layers · width 1024 · 16 heads · mlp 4096 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream. It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).

Usage (image → caption)

Needs the patched open_clip that provides create_vision_encoder_and_transforms (from https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow.

import json, numpy as np, torch
from PIL import Image
from huggingface_hub import hf_hub_download
from open_clip.factory import create_vision_encoder_and_transforms
from modeling_openvision2_decoder import OpenVision2TextDecoder, OpenVision2TextDecoderConfig
from safetensors.torch import load_file

repo = "UCSC-VLAA/openvision2-vit-huge-patch14-224-vision-only"
enc = create_vision_encoder_and_transforms(model_name=f"hf-hub:{repo}").eval()

cfg = json.load(open(hf_hub_download(repo, "text_decoder_config.json")))
dec = OpenVision2TextDecoder(OpenVision2TextDecoderConfig(
    width=cfg["width"], depth=cfg["depth"], num_heads=cfg["num_heads"], mlp_dim=cfg["mlp_dim"],
    vocab_size=cfg["vocab_size"], vision_width=cfg["vision_width"]))
dec.load_state_dict(load_file(hf_hub_download(repo, "caption_decoder.safetensors"))); dec.eval()
vocab = [l.rstrip("\n") for l in open(hf_hub_download(repo, "bert_base_vocab_bos_eos.txt"))]

def preprocess(path, res=224):
    im = Image.open(path).convert("RGB"); w, h = im.size; s = res / min(w, h)
    im = im.resize((round(w*s), round(h*s)), Image.BILINEAR); w, h = im.size
    l, t = (w-res)//2, (h-res)//2; im = im.crop((l, t, l+res, t+res))
    x = (np.asarray(im, np.float32) - np.array([.485,.456,.406])*255) / (np.array([.229,.224,.225])*255)
    return torch.tensor(x.transpose(2,0,1)[None], dtype=torch.float32)

with torch.no_grad():
    _, tokens = enc(preprocess("image.jpg"))
    ids = dec.generate(tokens, max_len=64, bos_id=1, eos_id=2)[0].tolist()

words = []
for i in ids:
    if i == 2: break              # eos
    if i in (0, 1, 2): continue   # pad / bos / eos
    tk = vocab[i]
    if tk.startswith("##"): words[-1] = words[-1] + tk[2:] if words else tk[2:]
    else: words.append(tk)
print(" ".join(words))

See caption_example.py for a runnable script.

Notes

• The encoder and decoder are a matched pair exported from the same training checkpoint.
• Captions are LLaVA-style dense descriptions (the decoder was trained on synthetic dense captions), so outputs are detailed and multi-sentence.