openvision2-vit-large-patch14-336-vision-only

UCSC-VLAA/openvision2-vit-large-patch14-336-vision-only

openvision2-vit-large-patch14-336-vision-only is a image-to-text model from UCSC-VLAA. Use it when you need a caption or text from an image. It is set up for open_clip.

This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image - a descriptive caption…

Downloads · 30 days

3% of all-time downloads

All-time downloads

1.3K

Public

Repo size

1.8 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin1.2 GB · 69%

At a glance

Task: Image-to-Text
Library: open_clip
Access: Public
Created: Sep 2, 2025
Updated: Aug 22, 2026
SHA: ef01023d

Model card

From the Hugging Face model README

OpenVision2 · L/14 @336 — vision encoder + caption decoder

This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image -> a descriptive caption. The encoder files are unchanged; only the decoder (and this card) were added.

Files

file	role
`open_clip_pytorch_model.bin`, `open_clip_config.json`	L/14 vision encoder (open_clip format)
`caption_decoder.safetensors`	caption text decoder weights
`text_decoder_config.json`	decoder architecture config
`modeling_openvision2_decoder.py`	standalone PyTorch decoder + `generate()`
`bert_base_vocab_bos_eos.txt`	tokenizer vocab (`[PAD]=0 [bos]=1 [eos]=2`)
`caption_example.py`	end-to-end image -> caption demo

Decoder architecture

A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder): ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and text is generated causally while attending to all image tokens.

12 layers · width 768 · 12 heads · mlp 3072 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream. It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).

Usage

Needs the patched open_clip providing create_vision_encoder_and_transforms (https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow. See caption_example.py:

python caption_example.py --image your.jpg

Notes

Encoder and decoder are a matched pair exported from the same training checkpoint.
Captions are LLaVA-style dense descriptions (multi-sentence, detailed).

Task: Image-to-Text
Library: open_clip
Created: Sep 2, 2025
Updated: Aug 22, 2026