VultronRetrieverCore-Qwen3.5-4.5B

VultronRetrieverCore-Qwen3.5-4.5B

#2 on ViDoRe V3 overall, behind only its 8B sibling, at 4.5B and 320 dimensions.

VultronRetrieverCore is the mid tier of the VultronRetriever family, a late-interaction (ColBERT-style) retriever that scores document pages directly from their rendered image: layout, tables, charts and text, across six languages. At 4.5B it ranks second on the full ViDoRe V3 board (only the 8B Prime scores higher) at a 320-dim index, narrower than most of the models around it.

The family has three tiers on one 320-dim recipe: the 8B Prime for maximum accuracy, the 4.5B Core (this model) for the accuracy/footprint mid-point, and the 0.8B Flash for latency- and footprint-sensitive serving. Trained and evaluated on Vultr Cloud.

Highlights

• #2 on ViDoRe V3: 63.57 mean nDCG@10 over all 10 tasks; the only model above it is the 8B Prime.
• #1 on V3 Nuclear: 54.90 nDCG@10, the top score on that private task across the whole board.
• Best V2 in its class: 66.12 nDCG@5, top of every ≤4.5B retriever.
• Official MTEB: V1 92.21, V2 66.12, V3 63.57.
• ~4.5B parameters, native 320-dim, ≈9 GB bf16.
• Six languages (en, fr, de, es, it, pt).

ViDoRe leaderboard (ranked by V3)

Ranked by ViDoRe V3 (mean nDCG@10), the headline benchmark; V1 and V2 are shown alongside. Our three tiers are in bold; models with a partial or missing V3 sit at the bottom.

V1/V2: full mean nDCG@5. V3: mean nDCG@10 over all 10 ViDoRe V3 tasks, the 8 public plus the 2 private tasks (Nuclear, Telecom) scored by the ViDoRe maintainers on held-out corpora. † = evaluated on the 8 public tasks only, not comparable to the 10-task means. Competitor figures: public MTEB ViDoRe leaderboard, 2026-07-04 snapshot; "—" = not reported on that benchmark.

At 4.5B, Core is #2 on the full V3 board; the only model above it is the 8B Prime, and on the private Nuclear task it posts the top score outright. On V2 it tops every ≤4.5B retriever. On the near-saturated V1 (the pack sits at 88–92) it scores 92.21, fourth on the board.

Official MTEB results (dim 320 / visual tokens 1792)

Per-task JSONs for the 8 public V3 tasks are in eval_results/; the 2 private tasks are scored by the ViDoRe maintainers and reported on the public leaderboard. Measured with the official MTEB late-interaction evaluator.

Why 320 dimensions

Late-interaction index size, memory footprint, and MaxSim scoring cost all scale with the embedding dimension. Core uses a native fixed 320-dim projection head (no Matryoshka truncation): a 320-dim index is a fraction of the size a 2048-4096-dim retriever carries, for proportionally lower storage, RAM, and query-time compute at serving scale, with no accuracy left on the table from prefix-truncating a wider head.

Intended use

• Visual document retrieval / multimodal RAG over PDFs, scans, slides and reports, including pages with layout, tables, charts and figures.
• Multilingual document collections (en, fr, de, es, it, pt).
• The accuracy/footprint mid-point of the family: more accurate than the 0.8B Flash tier, lighter than the 8B Prime flagship.

Out of scope: text-only semantic search, where a single-vector dense embedder is cheaper; generative QA (this is a retriever; pair it with a reader/LLM).

Method

Per-token MaxSim scoring captures fine-grained matches against tables, figures, and layout that a single-vector embedder averages away.

• Base: Qwen/Qwen3.5-4B (hybrid GatedDeltaNet + full-attention backbone).
• Late-interaction retriever (ColQwen3_5): native 320-dim multi-vector embeddings, MaxSim scoring, image + text inputs.
• Size: ~4.5B parameters, the generative head dropped for retrieval.
• Native 320-dim head (no Matryoshka): trained and operated at 320 dimensions directly.
• Model merging: five independently-seeded checkpoints (three in-batch ColBERT contrastive, two with mined hard negatives) merged per-block into one full-weight checkpoint. Per-block weights searched by TPE (explore) then BoTorch qLogNEI (exploit); the winner is the global argmax at the operating point.
• Trained at up to 1280 visual tokens, evaluated and deployed at 1792.

Training data

An enhanced, multilingual mixture of public and synthetic visual-document retrieval sources, spanning en, es, de, fr, it and pt, decontaminated against all three ViDoRe suites (V1/V2/V3): 0% measured overlap with the evaluation benchmarks. The training recipe and the assembled training dataset are not distributed in this repository.

Inputs and outputs

• Input: document-page images (RGB) and/or text queries; pages encode at up to 1792 visual tokens.
• Output: multi-vector embeddings, one 320-dim vector per token (not a single pooled vector).
• Scoring: late-interaction MaxSim between query-token and page-token vectors, via score_multi_vector.

Usage

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("vultr/VultronRetrieverCore-Qwen3.5-4.5B")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([23, 320]) torch.Size([1251, 320])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[16.2559, 10.9844],
#         [ 5.7363, 12.9688]], device='cuda:0')

ColPali Engine

The Qwen3.5 hybrid (GatedDeltaNet + full-attention) backbone has hard runtime kernel dependencies a vanilla ColQwen / PaliGemma card does not:

pip install "git+https://github.com/illuin-tech/colpali@2e0b927051af727238783af039dcc2c50a4d8c27"
pip install causal-conv1d flash-linear-attention

• causal-conv1d + flash-linear-attention are required (the hybrid layers import them at runtime).
• Use SDPA, matching the training and evaluation runtime.

import torch
from PIL import Image
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor

model = ColQwen3_5.from_pretrained(
    "vultr/VultronRetrieverCore-Qwen3.5-4.5B",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",   # required (see above)
    device_map="cuda:0",
).eval()
processor = ColQwen3_5Processor.from_pretrained(
    "vultr/VultronRetrieverCore-Qwen3.5-4.5B",
    max_num_visual_tokens=1792,
)

# Document pages (rendered to images) and text queries
images = [Image.open("page_0.png"), Image.open("page_1.png")]
queries = ["What was Q3 revenue?", "Summarize the safety findings."]

with torch.no_grad():
    doc_emb = model(**processor.process_images(images).to(model.device))
    qry_emb = model(**processor.process_queries(queries).to(model.device))

# Late-interaction MaxSim scoring (feed fp32 to match the eval discipline)
scores = processor.score_multi_vector(qry_emb.float(), doc_emb.float())
# scores[i, j] = relevance of query i to page j
print(scores.shape)  # torch.Size([2, 2])

config.json carries dim=320, so custom_text_proj is sized correctly at load, with no manual config edits needed. It also carries retrieval_attention_contract="causal"; a conflicting runtime override is an error.

vLLM

vLLM serves this model natively through its pooling runner (the ColQwen3_5 architecture), returning the per-token multi-vectors for late-interaction scoring. It requires a vLLM build that includes the projection-bias fix and explicit retrieval-attention metadata support. The handler reads retrieval_attention_contract="causal" from this checkpoint and constructs all eight full-attention layers as AttentionType.DECODER; it must reject missing, conflicting, or unsupported contract metadata. The server uses the stock chat/image processor, so the ColQwen3.5 prompt contract is applied client-side: wrap each page image in the instruction template, append the query-augmentation tokens to each query, and set the visual-token budget through mm-processor-kwargs. Prefix caching and chunked prefill remain off for the GatedDeltaNet hybrid.

pip install vllm

import torch
from PIL import Image
from vllm import LLM
MODEL = "vultr/VultronRetrieverCore-Qwen3.5-4.5B"
MAX_PIXELS = 1792 * 32 * 32   # max_num_visual_tokens * (patch_size 16 * merge_size 2)^2
llm = LLM(
    model=MODEL,
    runner="pooling",
    dtype="bfloat16",
    enable_prefix_caching=False,
    enable_chunked_prefill=False,
    mm_processor_kwargs={"min_pixels": 65536, "max_pixels": MAX_PIXELS},
)
# ColQwen3.5 processor contract, applied client-side:
IMAGE_PROMPT = ("<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
                "Describe the image.<|im_end|><|endoftext|>")
def query_prompt(q): return q + "<|endoftext|>" * 10   # query augmentation
images  = [Image.open("page_0.png"), Image.open("page_1.png")]
queries = ["What was Q3 revenue?", "Summarize the safety findings."]
doc_out = llm.encode([{"prompt": IMAGE_PROMPT, "multi_modal_data": {"image": im}}
                      for im in images], pooling_task="token_embed")
qry_out = llm.encode([query_prompt(q) for q in queries], pooling_task="token_embed")
def mv(o):  # one [num_tokens, 320] multi-vector per item, L2-normalized per token
    t = torch.as_tensor(o.outputs.data, dtype=torch.float32)
    return torch.nn.functional.normalize(t, p=2, dim=-1)
docs, qrys = [mv(o) for o in doc_out], [mv(o) for o in qry_out]
# late-interaction MaxSim: per query token take the best doc token, then sum
scores = [[(q @ d.T).max(dim=-1).values.sum().item() for d in docs] for q in qrys]
print(scores)  # scores[i][j] = relevance of query i to page j

To serve over HTTP instead:

vllm serve vultr/VultronRetrieverCore-Qwen3.5-4.5B \
  --runner pooling \
  --no-enable-prefix-caching --no-enable-chunked-prefill \
  --mm-processor-kwargs '{"min_pixels": 65536, "max_pixels": 1835008}'

Apply the same image template and query augmentation in your client requests. See the upstream example examples/pooling/score/colqwen3_5_rerank_online.py for the full online rerank flow.

Limitations

• Tuned for six languages (en, fr, de, es, it, pt); other languages are out of distribution.
• Late-interaction multi-vector indexes are larger than single-vector dense indexes: the trade for per-token layout/table/figure sensitivity (small for its class at 320-dim).
• This is the mid tier; for maximum ViDoRe V3 accuracy use the 8B flagship VultronRetrieverPrime-Qwen3.5-8B, and for the smallest footprint the 0.8B VultronRetrieverFlash-Qwen3.5-0.8B.

License

Apache 2.0, covering the contents of this repository: model weights, config, and evaluation results. Built on Qwen/Qwen3.5-4B (Apache 2.0); the upstream license and attribution are retained. The training recipe and the assembled training dataset are not distributed in this repository.

Citation

@misc{vultronretrievercore2026,
  title  = {VultronRetrieverCore-Qwen3.5-4.5B: Mid-Tier Late-Interaction Visual Document Retrieval at 320 Dimensions},
  author = {Georgiou, Athos (athrael-soju)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B}}
}

---

Trained and evaluated on Vultr Cloud.