jina-vlm

<p align="center"> <img src="https://huggingface.co/datasets/jinaai/documentation-images/resolve/main/logo.webp" alt="Jina AI: Your Search Foundation, Supercharged!" width="150px"> </p>

jina-vlm: Small Multilingual Vision Language Model

Blog | API | AWS | Azure | GCP | Arxiv

jina-vlm is a token-efficient 2.4B parameter vision-language model that achieves state-of-the-art multilingual VQA performance among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language decoder and makes use of image tiling and attention-pooling for token-efficient processing of arbitrary-resolution images.

Built on Qwen3-1.7B-Base with SigLIP2-So400M, it processes images via overlapping tiling with attention-based token pooling that reduces visual tokens by 4x while preserving spatial information. The model achieves the highest average score (72.3) across eight VQA benchmarks while leading on multilingual multimodal understanding (MMMB: 78.8, Multilingual MMBench: 74.3).

Via Jina API

We provide an OpenAI-compatible API at https://api-beta-vlm.jina.ai. All requests require a Jina API key in the Authorization header, get your API key at jina.ai.

Image from URL

curl https://api-beta-vlm.jina.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{
    "model": "jina-vlm",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]
    }]
  }'

Local image (base64)

curl https://api-beta-vlm.jina.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{
    "model": "jina-vlm",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'$(base64 -i image.jpg)'"}}
      ]
    }]
  }'

Text-only query

curl https://api-beta-vlm.jina.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{
    "model": "jina-vlm",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

Streaming response

Add "stream": true to receive tokens as they're generated:

curl https://api-beta-vlm.jina.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{
    "model": "jina-vlm",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a haiku about coding"}]
  }'

When the service is cold starting, you'll receive:

{
  "error": {
    "message": "Model is loading, please retry in 30-60 seconds. Cold start takes ~30s after the service scales up.",
    "code": 503
  }
}

Simply retry your request after waiting.

Local Installation

uv sync

For CUDA users with FlashAttention2 support:

uv sync --extra flash-attn

Using the CLI

You can directly chat with jina-vlm using the infer.py CLI:

# Single image
python infer.py -i image.jpg -p "What's in this image?"

# Streaming output
python infer.py -i image.jpg -p "Describe this image" --stream

# Multiple images
python infer.py -i img1.jpg -i img2.jpg -p "Compare these images"

# Text-only
python infer.py -p "What is the capital of France?"

Options:

• -m, --model: Model path. Auto-detects local repo (if config.json exists) or falls back to jinaai/jina-vlm from HuggingFace.
• -i, --image: Image path, URL, or glob pattern (can specify multiple times).
• -p, --prompt: Text prompt (can specify multiple times).
• --max-crops: Maximum crops (default: 12).
• --max-tokens: Maximum output tokens (default: 1024).
• --max-pixels: Max pixels per image, larger images are resized preserving aspect ratio.
• --stream: Enable streaming output.

Example:

python infer.py -i assets/the_persistence_of_memory.jpg -p "Describe this picture"

<table> <tr> <td width="40%"><b>Input</b></td> <td width="60%"><b>Output</b></td> </tr> <tr> <td><img src="./assets/the_persistence_of_memory.jpg" width="100%"></td> <td>

* Conversation 1/1
├── 🖼️Images: ['the_persistence_of_memory.jpg']
├── 📜Prompt: Describe this picture
└── 🧠Response: This image is a surreal painting
by Salvador Dalí, titled "The Persistence of
Memory." It features a dreamlike landscape with
a variety of melting clocks and other objects.
The central focus is a melting clock with a blue
face and yellow hands, which is hanging from a
branch...

Token usage: 1753 tokens (4.3%)
Generated in 8.68s | 20.04 tok/s

</td> </tr> </table>

Using Transformers

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig

processor = AutoProcessor.from_pretrained(
    'jinaai/jina-vlm', use_fast=False, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    'jinaai/jina-vlm',
    device_map='auto',
    trust_remote_code=True
)

image = 'https://picsum.photos/800/600'
conversation = [
    {
        'role': 'user',
        'content': [
            {'type': 'image', 'image': image},
            {'type': 'text', 'text': 'Describe this image'},
        ],
    }
]

text = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model.generate(
    **inputs,
    generation_config=GenerationConfig(max_new_tokens=512, do_sample=False),
    return_dict_in_generate=True,
    use_model_defaults=True,
)

response = processor.tokenizer.decode(
    output.sequences[0][inputs['input_ids'].shape[-1]:],
    skip_special_tokens=True
)
print(response)

<details> <summary>Multi-image inference</summary>

images = ['https://picsum.photos/id/1/800/600', 'https://picsum.photos/id/2/800/600']
conversation = [
    {
        'role': 'user',
        'content': [
            {'type': 'image', 'image': images[0]},
            {'type': 'image', 'image': images[1]},
            {'type': 'text', 'text': 'What is the difference between these images?'},
        ],
    }
]
text = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[text], images=images, padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model.generate(
    **inputs,
    generation_config=GenerationConfig(max_new_tokens=512, do_sample=False),
    return_dict_in_generate=True,
    use_model_defaults=True,
)
response = processor.tokenizer.decode(
    output.sequences[0][inputs['input_ids'].shape[-1]:],
    skip_special_tokens=True
)
print(response)

</details> <details> <summary>Text-only inference</summary>

conversation = [
    {
        'role': 'user',
        'content': [
            {'type': 'text', 'text': 'Explain quantum computing in simple terms'},
        ],
    }
]
text = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[text], padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model.generate(
    **inputs,
    generation_config=GenerationConfig(max_new_tokens=512, do_sample=False),
    return_dict_in_generate=True,
    use_model_defaults=True,
)
response = processor.tokenizer.decode(
    output.sequences[0][inputs['input_ids'].shape[-1]:],
    skip_special_tokens=True
)
print(response)

</details> <details> <summary>Batch inference</summary>

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig

processor = AutoProcessor.from_pretrained(
    'jinaai/jina-vlm', use_fast=False, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    'jinaai/jina-vlm',
    device_map='auto',
    torch_dtype=torch.bfloat16,
    attn_implementation='flash_attention_2',
    trust_remote_code=True
)

images = [
    'https://picsum.photos/id/22/800/600',
    'https://picsum.photos/id/49/800/600'
]
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[0]},
                {'type': 'text', 'text': 'What is the man doing in this image?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[1]},
                {'type': 'text', 'text': 'What country\'s flag is in this image?'},
            ],
        }
    ],
]

texts = processor.apply_chat_template(conversations, add_generation_prompt=True)
inputs = processor(text=texts, images=images, padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model.generate(
    **inputs,
    generation_config=GenerationConfig(max_new_tokens=512, do_sample=False),
    return_dict_in_generate=True,
    use_model_defaults=True,
)

for idx in range(len(output.sequences)):
    gen_ids = output.sequences[idx][inputs['input_ids'].shape[-1]:]
    response = processor.tokenizer.decode(gen_ids, skip_special_tokens=True)
    print(f"Response {idx+1}: {response}")

</details> <details> <summary>Batch inference with mixed examples</summary>

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig

processor = AutoProcessor.from_pretrained(
    'jinaai/jina-vlm', use_fast=False, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    'jinaai/jina-vlm',
    device_map='auto',
    torch_dtype=torch.bfloat16,
    attn_implementation='flash_attention_2',
    trust_remote_code=True
)

images = [
    ['https://picsum.photos/id/22/800/600'],
    ['https://picsum.photos/id/49/800/600'],
    ['https://picsum.photos/id/0/800/600', 'https://picsum.photos/id/2/800/600'],
    [],
]
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[0][0]},
                {'type': 'text', 'text': 'What is the man doing in this image?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[1][0]},
                {'type': 'text', 'text': 'What country\'s flag is in this image?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[2][0]},
                {'type': 'image', 'image': images[2][1]},
                {'type': 'text', 'text': 'What is the difference between these two images?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'Describe the concept of polymorphism in Computer Science'},
            ],
        }
    ],
]

texts = processor.apply_chat_template(conversations, add_generation_prompt=True)
inputs = processor(text=texts, images=images, padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model.generate(
    **inputs,
    generation_config=GenerationConfig(max_new_tokens=512, do_sample=False),
    return_dict_in_generate=True,
    use_model_defaults=True,
)

for idx in range(len(output.sequences)):
    gen_ids = output.sequences[idx][inputs['input_ids'].shape[-1]:]
    response = processor.tokenizer.decode(gen_ids, skip_special_tokens=True)
    print(f"Response {idx+1}: {response}")

</details> <details> <summary>Feature extraction</summary>

import torch
from transformers import AutoModel, AutoProcessor

processor = AutoProcessor.from_pretrained(
    'jinaai/jina-vlm', use_fast=False, trust_remote_code=True
)
model = AutoModel.from_pretrained(
    'jinaai/jina-vlm',
    device_map='auto',
    torch_dtype=torch.bfloat16,
    attn_implementation='flash_attention_2',
    trust_remote_code=True
)

images = [
    ['https://picsum.photos/id/22/800/600'],
    ['https://picsum.photos/id/49/800/600'],
    ['https://picsum.photos/id/0/800/600', 'https://picsum.photos/id/2/800/600'],
    [],
]
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[0][0]},
                {'type': 'text', 'text': 'What is the man doing in this image?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[1][0]},
                {'type': 'text', 'text': 'What country\'s flag is in this image?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'image', 'image': images[2][0]},
                {'type': 'image', 'image': images[2][1]},
                {'type': 'text', 'text': 'What is the difference between these two images?'},
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'Describe the concept of polymorphism in Computer Science'},
            ],
        }
    ],
]

texts = processor.apply_chat_template(conversations, add_generation_prompt=True)
inputs = processor(text=texts, images=images, padding='longest', return_tensors='pt')
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

output = model(**inputs)
print(output)

</details>

Using vLLM

The vLLM project and documentation.

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='generate',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/800/600'
                    }
                },
                {'type': 'text', 'text': 'Describe this image'}
            ],
        }
    ]
]
response = llm.chat(
    messages=conversations,
    add_generation_prompt=True,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
    sampling_params=SamplingParams(
        temperature=0.0,
        n=1,
        max_tokens=64,
        top_p=1.0,
        repetition_penalty=1.0,
        top_k=0,
    ),
)
print([r.outputs[0].text for r in response])

<details> <summary>Multi-image inference</summary>

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='generate',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/1/800/600'
                    }
                },
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/2/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the difference between these images?'}
            ],
        }
    ]
]
response = llm.chat(
    messages=conversations,
    add_generation_prompt=True,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
    sampling_params=SamplingParams(
        temperature=0.0,
        n=1,
        max_tokens=64,
        top_p=1.0,
        repetition_penalty=1.0,
        top_k=0,
    ),
)
print([r.outputs[0].text for r in response])

</details> <details> <summary>Text-only inference</summary>

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='generate',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'Explain quantum computing in simple terms'}
            ],
        }
    ]
]
response = llm.chat(
    messages=conversations,
    add_generation_prompt=True,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
    sampling_params=SamplingParams(
        temperature=0.0,
        n=1,
        max_tokens=64,
        top_p=1.0,
        repetition_penalty=1.0,
        top_k=0,
    ),
)
print([r.outputs[0].text for r in response])

</details> <details> <summary>Batch inference</summary>

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='generate',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/22/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the man doing in this image?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/49/800/600'
                    }
                },
                {'type': 'text', 'text': 'What country\'s flag is in this image?'}
            ],
        }
    ]
]
response = llm.chat(
    messages=conversations,
    add_generation_prompt=True,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
    sampling_params=SamplingParams(
        temperature=0.0,
        n=1,
        max_tokens=64,
        top_p=1.0,
        repetition_penalty=1.0,
        top_k=0,
    ),
)
print([r.outputs[0].text for r in response])

</details> <details> <summary>Batch inference with mixed examples</summary>

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='generate',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/22/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the man doing in this image?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/49/800/600'
                    }
                },
                {'type': 'text', 'text': 'What country\'s flag is in this image?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/0/800/600'
                    }
                },
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/2/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the difference between these two images?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'Describe the concept of polymorphism in Computer Science'}
            ],
        }
    ]
]
response = llm.chat(
    messages=conversations,
    add_generation_prompt=True,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
    sampling_params=SamplingParams(
        temperature=0.0,
        n=1,
        max_tokens=64,
        top_p=1.0,
        repetition_penalty=1.0,
        top_k=0,
    ),
)
print([r.outputs[0].text for r in response])

</details> <details> <summary>Feature extraction</summary>

from vllm import LLM, SamplingParams

llm = LLM(
    model='jinaai/jina-vlm',
    runner='pooling',
    trust_remote_code=True,
    dtype='bfloat16',  # or float16, float32
    hf_overrides={'_attn_implementation': 'flash_attention_2'},
)
conversations = [
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/22/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the man doing in this image?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/49/800/600'
                    }
                },
                {'type': 'text', 'text': 'What country\'s flag is in this image?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/0/800/600'
                    }
                },
                {
                    'type': 'image_url',
                    'image_url': {
                        'url': 'https://picsum.photos/id/2/800/600'
                    }
                },
                {'type': 'text', 'text': 'What is the difference between these two images?'}
            ],
        }
    ],
    [
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'Describe the concept of polymorphism in Computer Science'}
            ],
        }
    ]
]
prompts = llm.preprocess_chat(
    messages=conversations,
    chat_template_kwargs={
        'always_start_with_space': True,
        'image_prompt_token': '<|image|>',
    },
)
output = llm.encode(prompts, pooling_task='token_embed')
print([out.outputs.data for out in output])

</details>

Evaluation

Multilingual Understanding

General VQA Tasks

Text-Only Performance

Citation

If you find jina-vlm useful in your research, please cite our technical report:

@misc{koukounas2025jinavlm,
    title={Jina-VLM: Small Multilingual Vision Language Model},
    author={Andreas Koukounas and Georgios Mastrapas and Florian Hönicke and Sedigheh Eslami and Guillaume Roncari and Scott Martens and Han Xiao},
    year={2025},
    eprint={2512.04032},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2512.04032},
}

License

jina-vlm is licensed under CC BY-NC 4.0. For commercial usage inquiries, feel free to contact us.