colpali-v1.3-fp16

ColPali v1.3 ONNX (fp16)

[!IMPORTANT] This model is a Model Derivative of Gemma. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

Use of this model is subject to the use restrictions in Section 3.2 of the Gemma Terms of Use, including the Gemma Prohibited Use Policy. See License for details.

This repository contains an ONNX export of ColPali v1.3, converted by Qdrant for inference with FastEmbed.

• Original model: vidore/colpali-v1.3 by Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo (paper)
• Backbone: PaliGemma 3B by Google (paper)

Model Description

ColPali retrieves documents from images of their pages, with no OCR or layout parsing step. It encodes each page screenshot and each text query into a set of 128-dimensional vectors, one per image patch or query token, and scores a page against a query with ColBERT-style late interaction (MaxSim).

ColPali v1.3 is a LoRA fine-tune of PaliGemma 3B. This repository contains the adapter merged into its base model and exported as a single ONNX graph, so it runs without PyTorch or PEFT.

Model Details

• Architecture: PaliGemma (SigLIP vision encoder and Gemma 2B language model) with a 128-dimensional projection head
• Parameters: about 2.9B
• Embedding size: 128 per vector (multivector)
• Image input: 448x448 RGB; each page produces 1,030 vectors (1,024 image patches plus 6 prompt tokens)
• Query length: FastEmbed truncates queries to 50 tokens
• Format: ONNX (opset 14) with fp16 weights, stored in the external data file model.onnx_data (5.85 GB)
• Language: English

Usage

With FastEmbed

Requires FastEmbed 0.6.0 or later.

from fastembed import LateInteractionMultimodalEmbedding

model = LateInteractionMultimodalEmbedding(model_name="Qdrant/colpali-v1.3-fp16")

# Encode document pages (images)
page_embeddings = list(model.embed_image(["./page_1.png", "./page_2.png"]))
# one array of shape (1030, 128) per page

# Encode queries (text)
query_embeddings = list(model.embed_text(["What is Qdrant?"]))
# one array of shape (num_tokens, 128) per query

With Qdrant

Store each page as a multivector and let Qdrant compute MaxSim at query time:

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="pages",
    vectors_config=models.VectorParams(
        size=128,
        distance=models.Distance.COSINE,
        multivector_config=models.MultiVectorConfig(
            comparator=models.MultiVectorComparator.MAX_SIM
        ),
    ),
)

client.upsert(
    collection_name="pages",
    points=[
        models.PointStruct(id=i, vector=embedding.tolist())
        for i, embedding in enumerate(page_embeddings)
    ],
)

hits = client.query_points(
    collection_name="pages",
    query=query_embeddings[0].tolist(),
    limit=5,
)

With ONNX Runtime

The graph takes text and image inputs together:

It returns last_hidden_state as float16 with shape (batch, sequence, 128).

Images are resized to 448x448 (bicubic), scaled to [0, 1], and normalized with mean 0.5 and standard deviation 0.5. When embedding pages, the text input is 1,024 <image> tokens followed by a short fixed prompt. When embedding queries, pixel_values is an all-zero image. See FastEmbed's ColPali implementation for the exact token sequences.

Modifications

This repository contains modified files. In February 2025, Qdrant:

1. Merged the LoRA adapter from vidore/colpali-v1.3 into its base model, vidore/colpaligemma-3b-pt-448-base.
2. Exported the merged model to ONNX with fp16 weights (model.onnx, model.onnx_data) and wrote its config.json.
3. Re-saved tokenizer_config.json, which added an empty extra_special_tokens field.

tokenizer.json, special_tokens_map.json, and preprocessor_config.json are unmodified copies of the upstream files. The NOTICE file lists the same changes.

License

This model is a Model Derivative of Gemma and is distributed under the Gemma Terms of Use. A copy of the terms is in LICENSE, and the required notice is in NOTICE.

By using this model, you agree to the Gemma Terms of Use, including the use restrictions in Section 3.2 and the Gemma Prohibited Use Policy. If you distribute this model or a model derived from it, Section 3.1 requires you to pass the same terms, restrictions, and notice on to your recipients.

The ColPali v1.3 adapter weights were released by their authors under the MIT License, which is reproduced in NOTICE. That license covers the adapter on its own. It doesn't change the terms of the merged model in this repository.

Citation

If you use this model, please cite the ColPali paper:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models},
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449},
}

Acknowledgments

Thanks to the ColPali authors for releasing the model and training code, and to Google for PaliGemma. The ONNX conversion was done by the Qdrant team for use with FastEmbed.