llama-nemotron-embed-vl-1b-v2

Model Overview

Description:

llama-nemotron-embed-vl-1b-v2 was developed by NVIDIA for multimodal question-answering retrieval. The model can embed document pages in the form of image, text, or combined image–text inputs. Documents can be retrieved given a user query in text form. The model supports page images containing text, tables, charts, and infographics. We report the evaluation of this model on two internal multimodal retrieval benchmarks, and on the popular ViDoRe V1 and V2 benchmarks and the new Vidore V3 benchmark.

An embedding model is a crucial component of a retrieval system because it transforms information into dense vector representations. An embedding model is typically a transformer encoder that processes tokens of input text or images (for example, questions, passages, or page images) to output an embedding. llama-nemotron-embed-vl-1b-v2 is a combined language and vision model.

The llama-nemotron-embed-vl-1b-v2 is part of the Nemotron RAG collection of open models available on HuggingFace. It is also available for optimized inference as a NIM (NVIDIA Inference Microservice) from NVIDIA NeMo Retriever, which provides state-of-the-art, commercially-ready models and microservices optimized for the lowest latency and highest throughput. It features a production-ready information retrieval pipeline with enterprise support. The models that form the core of this solution have been trained using responsibly selected, auditable data sources. With multiple pre-trained models available as starting points, developers can readily customize them for domain-specific use cases, such as information technology, human resource help assistants, and research & development research assistants.

This model is ready for commercial use.

License/Terms of use

The use of this model is governed by the NVIDIA Open Model License Agreement and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama.

Deployment Geography:

Global <br>

Use Case: <br>

The llama-nemotron-embed-vl-1b-v2 is suitable for users who want to build a multimodal question-and-answer application over a large corpus, leveraging the latest dense retrieval technology.
The input of the model is a text or document image and the output a fixed-size embedding vector. The embedding model is a bi-encoder that supports context in textual format (e.g. the query or the OCR text of a page or a section of a document) or the image of a document page.
Typically, the embedding model is used first to embed (vectorize) the whole corpus (document images or text chunks), and embeddings are stored stored in a vector database associated to its raw content (image or text). Then at inference time, the embedding model is used to embed the query. The embeddings of the query and relevant context from the corpus should be close in the embedding space.

Release Date: <br>

12/18/2025 via https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2

References(s):

Technical report - "Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model" <br>

Citation

@inproceedings{moreira2025_nvretriever,
    author = {Moreira, Gabriel de Souza P. and Osmulski, Radek and Xu, Mengyao and Ak, Ronay and Schifferer, Benedikt and Oldridge, Even},
    title = {Improving Text Embedding Models with Positive-aware Hard-negative Mining},
    year = {2025},
    isbn = {9798400720406},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3746252.3761254},
    doi = {10.1145/3746252.3761254},
    pages = {2169–2178},
    numpages = {10},
    keywords = {contrastive learning, distillation, embedding models, hard-negative mining, rag, text retrieval, transformers},
    location = {Seoul, Republic of Korea},
    series = {CIKM '25}
}

Model Architecture

Architecture Type: Transformer <br>

Network Architecture: Eagle VLM architecture with Llama 3.2 1B language model and SigLip2 400 image encoder<br>

The llama-nemotron-embed-vl-1b-v2 embedding model is a transformer encoder, with approximately 1.7B parameters. It is a fine-tuned version of NVIDIA Eagle family of models, using Llama 3.2 1B language model and SigLip2 400M image encoder. The language model submodule has 16 layers with embedding size of 2048, and is pre-trained on public datasets. Embedding models for retrieval are typically trained with a bi-encoder architecture, that encodes query and document independently. The model applies mean pooling over the output token embeddings from the language model, so that it outputs a single embedding with 2048 dimensions. Contrastive learning is used to train the embedding model to maximize the similarity between the query and the document page that contains the answer, while minimizing the similarity between the query and sampled negative pages that are not useful to answer the question.

The vision-language model encoder incorporates key innovations from NVIDIA, including Eagle 2 research and nemoretriever-parse, which use a tiling-based VLM architecture. This architecture, available on Hugging Face, significantly enhances multimodal understanding through its dynamic tiling and mixture of vision encoders design. It particularly improves performance on tasks that involve high-resolution images and complex visual content.

Number of model parameters:

• Llama 3.2 1B language model: 1.23 B (Transformer parameters: 973 M, Token embedding parameters: 262 M)
• SigLip 2 image encoder: 428.77 M

Input(s): <br>

Input Type(s): Image, Text <br>

Input Format(s): <br>

• Image: Red, Green, Blue (RGB) <br>
• Text: String <br>

Input Parameters: <br>

• Image: Two-Dimensional (2D) <br>
• Text: One-Dimensional (1D) <br>

Output Parameters: <br>

• Image/Text Embedding (2D) - embedding of 2048 dimensions <br>

Other Properties Related to Input:

• The model's maximum context length we evaluated is 10240 tokens. <br>
• Each image tile consumes 256 tokens. We have tested this model extensively with these settings on config.json - max_input_tiles = 6, use_thumbnails = True, so that every image is split into maximum 6 tiles + 1 thumbnail (whole image at lower resolution), consuming about 1792 visual tokens. If you embed both page image and text (e.g. page OCR), the sum of the visual tokens (explained above) and the text tokens should not be higher than 10240 tokens.

Output(s)

Output Type: Floats<br> Output Format: List of float arrays (embeddings) <br> Output: Model outputs embedding vectors of maximum dimension 2048 for each input.<br> Other Properties Related to Output: N/A<br>

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (such as GPU cores) and software frameworks (such as CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Installation

The model requires transformers>=4.56.0 and optionally flash-attention.

pip install "transformers>=4.56.0"
pip install "flash-attn>=2.6.3,<2.8" --no-build-isolation

Sentence Transformers Usage

Install Sentence Transformers:

pip install sentence_transformers

This model supports text, image, and image+text embedding with Sentence Transformers.

from sentence_transformers import SentenceTransformer
from transformers.image_utils import load_image

model = SentenceTransformer("nvidia/llama-nemotron-embed-vl-1b-v2", trust_remote_code=True)

query = "How is AI improving the intelligence and capabilities of robots?"
documents = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "AI is transforming autonomous vehicles by enabling safer, smarter, and more reliable decision-making on the road.",
    "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences.",
]
# Images can be URLs, file paths, or PIL Images
images = [
    load_image("https://developer.download.nvidia.com/images/isaac/nvidia-isaac-lab-1920x1080.jpg"),
    load_image("https://blogs.nvidia.com/wp-content/uploads/2018/01/automotive-key-visual-corp-blog-level4-av-og-1280x680-1.png"),
    load_image("https://developer-blogs.nvidia.com/wp-content/uploads/2025/02/hc-press-evo2-nim-25-featured-b.jpg"),
]

# Text-only encoding
query_embeddings = model.encode_query([query])
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (1, 2048) (3, 2048)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4142, 0.4046, 0.0421]])

# Image-only encoding
image_embeddings = model.encode(images)
print(image_embeddings.shape)
# (3, 2048)

similarities = model.similarity(query_embeddings, image_embeddings)
print(similarities)
# tensor([[ 0.2766,  0.1378, -0.0062]])

# Image+text encoding (e.g. page image + OCR text)
multimodal_docs = [{"image": img, "text": txt} for img, txt in zip(images, documents)]
multimodal_embeddings = model.encode(multimodal_docs)
print(multimodal_embeddings.shape)
# (3, 2048)

similarities = model.similarity(query_embeddings, multimodal_embeddings)
print(similarities)
# tensor([[0.3385, 0.2688, 0.0077]])

Transformers Usage

import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

modality = "image"
# Load model
model_name_or_path = "nvidia/llama-nemotron-embed-vl-1b-v2"

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
    device_map="auto"
).eval()


# Set max number of tokens (p_max_length) based on modality
if modality == "image":
    p_max_length = 2048
elif modality == "image_text":
    p_max_length = 10240
elif modality == "text":
    p_max_length = 8192
model.processor.p_max_length = p_max_length
# Sets max number of tiles an image can be split. Each tile consumes 256 tokens.
model.processor.max_input_tiles = 6
# Enables an extra tile with the full image at lower resolution
model.processor.use_thumbnail = True


# Example usage: single query with multiple image documents
query = "How is AI improving the intelligence and capabilities of robots?"
image_paths = [
    "https://developer.download.nvidia.com/images/isaac/nvidia-isaac-lab-1920x1080.jpg",
    "https://blogs.nvidia.com/wp-content/uploads/2018/01/automotive-key-visual-corp-blog-level4-av-og-1280x680-1.png",
    "https://developer-blogs.nvidia.com/wp-content/uploads/2025/02/hc-press-evo2-nim-25-featured-b.jpg"
]

# Load all images (load_image handles both local paths and URLs)
images = [load_image(img_path) for img_path in image_paths]

# Text descriptions corresponding to each image/document (used in image_text and text modalities)
document_texts = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "AI is transforming autonomous vehicles by enabling safer, smarter, and more reliable decision-making on the road.",
    "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences."
]

# Run inference (common for all modalities)
with torch.inference_mode():
    queries_embeddings = model.encode_queries([query])
    
    if modality == "image_text":
        documents_embeddings = model.encode_documents(images=images, texts=document_texts)
    elif modality == "image":
        documents_embeddings = model.encode_documents(images=images)
    elif modality == "text":
        documents_embeddings = model.encode_documents(texts=document_texts)

def _l2_normalize(x: torch.Tensor, eps: float = 1e-12) -> torch.Tensor:
    return x / (x.norm(p=2, dim=-1, keepdim=True) + eps)      

# Computes cosine similarity (as they are already normalized) between the query embeddings and the document embeddings
cos_sim = _l2_normalize(queries_embeddings) @ _l2_normalize(documents_embeddings).T

# Flatten logits to 1D array (handle both [batch_size] and [batch_size, 1] shapes)
cos_sim_flat = cos_sim.flatten()
    
# Get sorted indices (highest to lowest)
sorted_indices = torch.argsort(cos_sim_flat, descending=True)

print(f"\nQuery: {query}\n")
print(f"\nRanking (highest to lowest relevance for the modality {modality}):")
for rank, idx in enumerate(sorted_indices, 1):
    doc_idx = idx.item()
    sim_val = cos_sim_flat[doc_idx].item()
    if modality == "text":
        print(f"  Rank {rank}: cos_sim={sim_val:.4f} | Text: {document_texts[doc_idx]}")
    else:  # image or image_text modality
        print(f"  Rank {rank}: cos_sim={sim_val:.4f} | Image: {image_paths[doc_idx]}")

Software Integration:

Runtime Engine(s): TensorRT, Triton, NeMo Retriever Embedding NIM (upcoming)<br> Supported Hardware Microarchitecture Compatibility: NVIDIA Ampere, NVIDIA Blackwell, NVIDIA Hopper, NVIDIA Lovelace<br> Preferred/Supported Operating System(s): Linux<br>

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>

Model Version(s):

llama-nemotron-embed-vl-1b-v2<br>

Training and Evaluation Datasets:

Training Dataset

The development of large-scale, public, open-QA datasets has enabled tremendous progress in powerful embedding models. However, the following issues limit the use of these models in commercial settings.

• One popular dataset, named MS MARCO, restricts ‌commercial licensing.
• Many multimodal datasets use synthetic data generation with proprietary models.

NVIDIA's training dataset is based on public QA datasets, and only includes datasets that have a license for commercial applications.

Properties: The text component is comprised of semi-supervised pre-training on 12M samples from public datasets and fine-tuning on 1.5M samples from public datasets. The VLM component uses only commercially-viable data from the Eagle2 training data and other public datasets.<br>

Data Modality: Image, Text

Image Training Data Size

• 1 Million to 1 Billion Images (about 2,57 Million)

Text Training Data Size

• 1 Billion to 10 Trillion Tokens (about 1.6 Billion)

Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic<br> Labeling Method by dataset: Hybrid: Automated, Human, Synthetic<br>

Evaluation Datasets

Vision document retrieval benchmarks

We evaluated llama-nemotron-embed-vl-1b-v2 on the popular ViDoRe V1, V2 and on the new ViDoRe V3.
More details on ViDoRe leaderboard can be found on their leaderboard.

We also evaluated the llama-nemotron-embed-vl-1b-v2 on two internal visual document retrieval datasets:

• DigitalCorpora-10k: A dataset with questions based on a corpus of 10k documents from DigitalCorpora that have a good mixture of text, tables, and charts.
• Earnings V2: an internal retrieval dataset of 287 questions based on 500 PDFs, mostly consisting of earnings reports from big tech companies.

For those interested in reproducing our results, one of our internal datasets (DigitalCorpora-10k) can be created by following instructions in this notebook (download script) from the NeMo Retriever Extraction GitHub repository.

Text retrieval benchmarks

We evaluated llama-nemotron-embed-vl-1b-v2 on 92 text retrieval datasets, from the benchmarks BEIR, MIRACL (multi-language), MLQA (cross-language) and MLDR (long-context).

Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic<br> Labeling Method by dataset: Hybrid: Automated, Human, Synthetic<br>

Evaluation Results

Visual Document Retrieval (page retrieval)

In this section, we compare the performance of llama-nemotron-embed-vl-1b-v2 with its previous version llama-3.2-nemoretriever-1b-vl-embed-v1 (closed weights), available as a NIM here. You can see here how the previous model compares to other small sized VLMs.

In below table, it is possible to see that the new llama-nemotron-embed-vl-1b-v2 provides much better retrieval accuracy (Recall@5) for the image and image+text modalities than its predecessor.
Note: Image+Text modality means that both the page image and its text (that might be extracted by some OCR library like NV-Ingest) are fed as input to the embedding model for more accurate representation and retrieval.

<table> <caption><b><em>Visual Document Retrieval benchmarks (page retrieval) - Avg Recall@5 on DC10k, Earnings V2, ViDoRe V1, V2, V3</em></b></caption> <thead> <tr> <th></th> <th colspan="3">Modality</th> </tr></thead> <tbody> <tr> <td>Model</td> <td>Text</td> <td>Image</td> <td>Image + Text</td> </tr> <tr> <td>llama-nemotron-embed-1b-v2 (former name: llama-3_2-nv-embedqa-1b-v2)</td> <td>69.35%</td> <td>-</td> <td>-</td> </tr> <tr> <td>llama-3.2-nemoretriever-1b-vlm-embed-v1 (closed weights, NIM-only)</td> <td>71.07%</td> <td>70.46%</td> <td>71.71%</td> </tr> <tr> <td><b>llama-nemotron-embed-vl-1b-v2</b></td> <td>71.04%</td> <td><b>71.20%</b></td> <td><b>73.24%</b></td> </tr> </tbody> </table>

Text Retrieval benchmarks (chunk retrieval)

The llama-nemotron-embed-vl-1b-v2 also improves retrieval accuracy on text retrieval benchmarks compared to our competitive text-only embedding model llama-nemotron-embed-1b-v2. That means you can deploy our single VLM-based model llama-nemotron-embed-vl-1b-v2 regardless the modality of your corpus to be retrieved is image or text.

<table> <caption><b><em>Text Retrieval benchmarks (chunk retrieval) - Avg. Recall@5</em></b></caption> <thead> <tr> <th><b>Model<b/></th> <th><b>BEIR retrieval + TechQA</b></th> <th><b>MIRACL</b></th> <th><b>MLQA</b></th> <th><b>MLDR</b></th> <th><em>Average</em></th> </tr></thead> <tbody> <tr> <td>llama-nemotron-embed-1b-v2 (former name: llama-3_2-nv-embedqa-1b-v2)</td> <td>68.60%</td> <td>60.75%</td> <td>79.86%</td> <td>59.55%</td> <td>67.19%</td> </tr> <tr> <td><b>llama-nemotron-embed-vl-1b-v2</b></td> <td><b>69.19%</b></td> <td>60.48%</td> <td><b>79.90%</b></td> <td><b>60.09%</b></td> <td><b>67.42%</b></td> </tr> </tbody> </table>

Inference

Acceleration Engine: TensorRT <br> Test Hardware: H100, A100, L40S, A10G, B200, RTX PRO 6000

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

For more detailed information on ethical considerations for this model, see the Model Card++ tab for the Explainability, Bias, Safety & Security, and Privacy subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>

Bias

Explainability

Privacy

Safety