Ovis-Omni-Embedding-3B-boundary-bf16-w2-w4

<!-- <a href="https://github.com/ATH-MaaS/Ovis-Omni-Embedding" target="_blank" style="margin: 2px;"> <img alt="GitHub" src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" style="display: inline-block; vertical-align: middle;"/> </a> -->

Ovis-Omni-Embedding-3B

Experimental vLLM mixed-precision derivative

This repository is a model-free RTN compressed-tensors quantization of the original Ovis-Omni-Embedding-3B model, published separately as pt810/Ovis-Omni-Embedding-3B-boundary-bf16-w2-w4.

Layer allocation: layers 0-1 remain BF16, layers 2-21 use W2A16, and layers 22-27 use W4A16. The multimodal towers and projection components remain in their original precision. This arrangement is intended for vLLM's compressed-tensors loader; it is not a GPTQ or BitsAndBytes checkpoint.

Validated with vLLM 0.30.0 on an 8 GiB GPU using --max-model-len 32768 --max-num-seqs 1 --max-num-batched-tokens 512 --kv-cache-dtype fp8 --cpu-offload-gb 2 --gpu-memory-utilization 0.80 --enforce-eager. A 32,701 token text request returned a finite, unit-normalized 2,048-dimensional embedding. The 1,024-dimensional deployment variant is provided by the pooler configuration in the Docker image.

This is an experimental accuracy/speed baseline. Compare embeddings against the original model before production use.

<div align="center"> <img src="./ovis_logo.png" width="30%"/> </div> <p align="center"> <a href="https://github.com/ATH-MaaS/Ovis-Omni-Embedding"><img src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" alt="github"></a> <a href="https://huggingface.co/ATH-MaaS/Ovis-Omni-Embedding-3B"><img src="https://img.shields.io/badge/🤗_Model_Page-Ovis--Omni--Embedding--3B-yellow" alt="model page"></a> <a href="https://arxiv.org/pdf/2609.25165"><img src="https://img.shields.io/badge/📖_Technical_Report-arXiv-b31b1b.svg" alt="technical report"></a> </p>

Ovis-Omni-Embedding-3B is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.

The model is initialized from Qwen2.5-Omni-3B. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.

Ovis-Omni-Embedding-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.

Architecture

<p align="center"> <img src="./ovis_embedding_model_architecture.png" alt="Architecture of Ovis-Omni-Embedding-3B and Ovis-Embedding-VL-2B/9B" width="100%"/> </p>

Figure 2 from the technical report. Ovis-Omni-Embedding-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.

An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.

Training

<p align="center"> <img src="./ovis_embedding_data_centric.png" alt="Broad omni-modal training data and homogeneous-source sampling" width="100%"/> </p>

Figure 4 from the technical report. The training corpus spans text, images, video, audio, visual documents, and interleaved inputs. Homogeneous-source sampling forms each micro-batch from one dataset and deduplicates pooled candidates to create informative in-batch negatives without positive collisions.

Training follows three stages:

1. Omni-modal contrastive pretraining. Globally mixed candidates and cross-device in-batch negatives establish broad alignment. The objective combines difficulty-aware focal contrastive learning with similarity-distribution distillation from complementary modality experts.
2. Full-parameter homogeneous finetuning. High-quality downstream data refine fine-grained discrimination. Samples within each micro-batch come from one dataset, and candidate deduplication prevents positive collisions and false in-batch negatives.
3. Annealing Embedding Distillation. Teacher-correct examples are retained, unresolved student examples are upsampled, and confidence-adaptive forward-KL supervision transfers complementary expert capabilities without adding inference-time towers.

<p align="center"> <img src="./ovis_embedding_train_inference.png" alt="Embedding Distillation and inference-time low-rank decomposition" width="100%"/> </p>

Figure 5 from the technical report. Embedding Distillation transfers similarity distributions from complementary experts, while the inference-time low-rank module combines a shared PCA basis with lightweight residual adapters for compact embeddings.

Retrieval interface

Ovis-Omni-Embedding-3B is a bi-encoder, not a cross-encoder:

1. Pair the query with the task instruction and format it with the model's native processor and chat template.
2. Encode queries and candidates independently.
3. Extract the final-layer hidden state at the last non-padding token.
4. L2-normalize the query and candidate embeddings.
5. Rank candidates by cosine similarity, equivalently the dot product of the normalized vectors.

No answer generation or query-candidate cross-attention is used during retrieval. Classification labels, passages, images, documents, videos, audio clips, and multimodal items are all treated as candidates in the same embedding space.

Performance

MMEB-v3

MMEB-v3 is an omni-modal benchmark comprising 190 datasets across image, video, visual-document, text, audio, and agent retrieval. Ovis-Omni-Embedding-3B achieves 58.46 overall, outperforming the strongest compared baseline by 5.19 points, and ranks first on the aggregate score of every modality group.

Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Omni-Embedding-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.

<p align="center"> <img src="./ovis_blog_table1.png" alt="Complete MMEB-v3 comparison for Ovis-Omni-Embedding-3B and four omni-modal embedding baselines" width="100%"/> </p>

Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.

Additional benchmark results

These benchmark families use their own official aggregation procedures, so their scores should not be averaged together. MAEB and MVEB results are local evaluations inserted into the corresponding leaderboard snapshots, as described in the technical report.

Embedding dimensions

The native output width is 2048 because no embedding projection head is added to the backbone. For deployments with tighter storage or latency budgets, the post-hoc elastic-dimension module supports 1024, 512, 256, and 128 dimensions. It combines a shared, modality-balanced PCA rotation with a zero-initialized residual linear adapter and folds both operations into one projection matrix at inference time.

Always use the same dimensionality and transformation for queries and candidates, and L2-normalize after projection.

Intended use

The model is intended for embedding extraction and retrieval over supported unimodal or interleaved multimodal content, including:

• semantic and cross-modal search;
• multimodal RAG indexing and retrieval;
• image and visual-document retrieval;
• video and audio retrieval;
• recommendation and nearest-neighbor matching;
• retrieval of tools, GUI states, and memory for agents.

Limitations

• This checkpoint produces retrieval embeddings; it is not intended as a speech or text generation model.
• Retrieval quality depends on using task-appropriate query instructions and the model's native preprocessing and chat template.
• Performance varies by task. In the reported MMEB-v3 comparison, MultiConIR is the principal weakness, while several established visual-document suites and memory retrieval remain below the best specialist result.
• Benchmark scores may not directly predict performance on a new domain. Evaluate with representative queries, candidates, and retrieval metrics before deployment.
• For high-stakes applications, embeddings should be combined with domain-specific evaluation, access controls, and, where appropriate, a second-stage reranker.

License

This model is released under the Apache 2.0 license.

Resources

• Ovis-Omni-Embedding GitHub repository
• Ovis-VL-Embedding GitHub repository

This repository contains the weights for Ovis-Omni-Embedding-3B.

Independent screening and fidelity status

This experimental boundary layout passed one 32,701-token vLLM embedding request with finite unit-normalized output. That is a context/serving result, not a fidelity result. Cosine drift, nearest-neighbor agreement, and Recall@K must be measured against persisted BF16 reference vectors from the same fixture; no production-quality claim is made without that comparison.

The completed 60-case BF16 comparison measured mean cosine 0.25641491, mean drift 0.74358509, maximum drift 0.99829924, nearest-neighbor agreement 15%, Recall@1 15%, Recall@5 48.33%, Recall@10 100%, and nDCG 0.46318916. This boundary layout is therefore a serving/context experiment, not a quality-preserving release.