NeoMME-800M-Retriever-ST-late

<p align="left"> <img src="https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true" alt="NeoMME logo" style="max-height: 140px;"> </p>

NeoMME-Retriever (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder 🔎

[!IMPORTANT] NeoMME-Retriever (800M) variants:

• Default (transformers): Returns dense and multi-vector embeddings together with a single forward pass. Recommended for most use cases and inference.
• ST dense: Supports independent dense fine-tuning with Sentence Transformers.
• ST late-interaction [current]: Supports independent multi-vector fine-tuning with Sentence Transformers.

NeoMME-800M-Retriever-ST-late is a model for multimodal document retrieval. Fine-tuned from NeoMME-800M, it encodes text queries and documents (text or page screenshots) using one shared bidirectional Transformer encoder.

This model can be used with Sentence Transformers, but can only generate multi-vector embeddings.

<table> <thead> <tr style="background-color: rgba(146, 81, 247, 0.20);"><th>Specification</th><th>Value</th></tr> </thead> <tbody> <tr><td>Parameters</td><td>800M</td></tr> <tr><td>Vocabulary</td><td>131,072 tokens</td></tr> <tr><td>Context length</td><td>16,384 tokens</td></tr> <tr><td>Hidden size</td><td>1,792</td></tr> <tr><td>Image patches</td><td>32 × 32 pixels, up to 2,048 pixels on the longest side (default)</td></tr> <tr><td>Multi-vector embeddings</td><td>128 dimensions per text token or image patch</td></tr> </tbody> </table>

Multi-vector embeddings are L2-normalized and use MeanMaxSim scoring.

Performance

All scores use the metric shown at the full trained dimensions. Higher is better. ViDoRe v3, v2, and v1 measure visual document retrieval, while BEIR-15 measures text retrieval.

<table> <thead> <tr><th rowspan="2">Benchmark</th><th rowspan="2">Metric</th><th colspan="2">NeoMME-260M</th><th colspan="2" style="background-color: rgba(37, 99, 235, 0.20);">NeoMME-800M</th></tr> <tr><th>Late interaction</th><th>Dense</th><th style="background-color: rgba(37, 99, 235, 0.20);">Late interaction [current]</th><th>Dense</th></tr> </thead> <tbody> <tr><td>ViDoRe v3</td><td>nDCG@10</td><td>0.5226</td><td>0.3907</td><td style="background-color: rgba(37, 99, 235, 0.20);">0.5560</td><td>0.4391</td></tr> <tr><td>ViDoRe v2</td><td>nDCG@5</td><td>0.5218</td><td>0.4075</td><td style="background-color: rgba(37, 99, 235, 0.20);">0.5591</td><td>0.4475</td></tr> <tr><td>ViDoRe v1</td><td>nDCG@5</td><td>0.8598</td><td>0.7552</td><td style="background-color: rgba(37, 99, 235, 0.20);">0.8744</td><td>0.7993</td></tr> <tr><td>BEIR-15</td><td>nDCG@10</td><td>0.4881</td><td>0.3055</td><td style="background-color: rgba(37, 99, 235, 0.20);">0.5126</td><td>0.3686</td></tr> </tbody> </table>

Usage

pip install -U "sentence-transformers[image]"

from sentence_transformers import MultiVectorEncoder


model = MultiVectorEncoder("Hcompany/NeoMME-800M-Retriever-ST-late")

queries = [
    "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?",
    "Which hour of the day had the highest overall electricity generation in 2019?",
]
image_documents = [
    "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/shift_kazakhstan.jpg?raw=true",
    "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/energy_electricity_generation.jpg?raw=true",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(image_documents)
late_scores = model.similarity(query_embeddings, document_embeddings)

# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].
print(late_scores)

The score tensor has shape (num_queries, num_documents) and late_scores[i, j] is the score between query i and document j. A larger value indicates a closer match.

Text documents use the same API:

text_documents = [
    "Kazakhstan produces oil from both onshore and offshore fields.",
    "Electricity generation varies throughout the day.",
]
text_document_embeddings = model.encode_document(text_documents)

The Transformers repository returns dense and multi-vector embeddings together. Use it directly when both representations are needed from one backbone pass.

Training

NeoMME-800M-Retriever was fine-tuned from NeoMME-800M on text retrieval and document-page images. Training uses a joint late-interaction and Matryoshka dense contrastive objective.

The NeoMME technical report describes the full fine-tuning recipe.

Limitations

With Sentence Transformers, only one of the two retrieval heads can be used at a time.

License

Model weights are released under the Apache 2.0 license.

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}