openvision2-vit-huge-patch14-336-vision-only

UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only

openvision2-vit-huge-patch14-336-vision-only is a image-to-text model from UCSC-VLAA. Use it when you need a caption or text from an image. It is set up for open_clip.

This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image - a descriptive caption…

Downloads · 30 days

5% of all-time downloads

All-time downloads

135

Public

Repo size

4 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin2.5 GB · 63%

At a glance

Task: Image-to-Text
Library: open_clip
Access: Public
Created: Sep 2, 2025
Updated: Aug 22, 2026
SHA: 0bda68f7

Model card

From the Hugging Face model README

OpenVision2 · H/14 @336 — vision encoder + caption decoder

This repo now contains the full OpenVision2 generative model: the vision encoder (originally released here) plus the caption text decoder it was jointly trained with. Together they map an image -> a descriptive caption. The encoder files are unchanged; only the decoder (and this card) were added.

Files

file	role
`open_clip_pytorch_model.bin`, `open_clip_config.json`	H/14 vision encoder (open_clip format)
`caption_decoder.safetensors`	caption text decoder weights
`text_decoder_config.json`	decoder architecture config
`modeling_openvision2_decoder.py`	standalone PyTorch decoder + `generate()`
`bert_base_vocab_bos_eos.txt`	tokenizer vocab (`[PAD]=0 [bos]=1 [eos]=2`)
`caption_example.py`	end-to-end image -> caption demo

Decoder architecture

A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder): ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and text is generated causally while attending to all image tokens.

24 layers · width 1024 · 16 heads · mlp 4096 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream. It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).

Usage

Needs the patched open_clip providing create_vision_encoder_and_transforms (https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow. See caption_example.py:

python caption_example.py --image your.jpg

Notes

Encoder and decoder are a matched pair exported from the same training checkpoint.
Captions are LLaVA-style dense descriptions (multi-sentence, detailed).

Task: Image-to-Text
Library: open_clip
Created: Sep 2, 2025
Updated: Aug 22, 2026