Ovis-Omni-Embedding-3B-bnb-8bit

pt810/Ovis-Omni-Embedding-3B-bnb-8bit

Ovis-Omni-Embedding-3B-bnb-8bit is a feature extraction model from pt810. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as apache-2.0.

This repository is a quantized derivative of the original model:

Downloads · 30 days

32% of all-time downloads

All-time downloads

Public

Parameters

5.5B

6.3 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors6.2 GB · 100%

Parameter types

How the weights are stored.

I85.1B · 92%

Model card

From the Hugging Face model README

Quantized derivative

This repository is a quantized derivative of the original model:

Original model: ATH-MaaS/Ovis-Omni-Embedding-3B
Source revision used: 926092ea607bdba164c9985a22a2b6130d8b6118
Quantization: BitsAndBytes 8-bit (load_in_8bit=True)
Native hidden width: 2048
Validated output width: 1024

The model was reloaded with Transformers and produced a normalized embedding with shape [1, 1024]. The 1024-dimensional result currently takes the first 1024 hidden coordinates; it is not the upstream learned elastic projection.

See QUANTIZATION.md for conversion and validation details.

vLLM status

vLLM recognizes the declared Qwen2_5OmniForConditionalGeneration architecture and the pooling conversion path (--runner pooling --convert embed). However, this 8-bit artifact currently fails before model loading because standard vLLM rejects its BitsAndBytes quantization_config with Unknown quantization method: bitsandbytes. It therefore has not been shown to serve successfully. Use the Transformers loading path described below, or convert to a vLLM-supported quantization format first.

Ovis-Omni-Embedding-3B

Ovis-Omni-Embedding-3B is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.

The model is initialized from Qwen2.5-Omni-3B. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.

Ovis-Omni-Embedding-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.

Architecture

Figure 2 from the technical report. Ovis-Omni-Embedding-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.

An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.

Training

Figure 4 from the technical report. The training corpus spans text, images, video, audio, visual documents, and interleaved inputs. Homogeneous-source sampling forms each micro-batch from one dataset and deduplicates pooled candidates to create informative in-batch negatives without positive collisions.

Training follows three stages:

Omni-modal contrastive pretraining. Globally mixed candidates and cross-device in-batch negatives establish broad alignment. The objective combines difficulty-aware focal contrastive learning with similarity-distribution distillation from complementary modality experts.
Full-parameter homogeneous finetuning. High-quality downstream data refine fine-grained discrimination. Samples within each micro-batch come from one dataset, and candidate deduplication prevents positive collisions and false in-batch negatives.
Annealing Embedding Distillation. Teacher-correct examples are retained, unresolved student examples are upsampled, and confidence-adaptive forward-KL supervision transfers complementary expert capabilities without adding inference-time towers.

Figure 5 from the technical report. Embedding Distillation transfers similarity distributions from complementary experts, while the inference-time low-rank module combines a shared PCA basis with lightweight residual adapters for compact embeddings.

Retrieval interface

Ovis-Omni-Embedding-3B is a bi-encoder, not a cross-encoder:

Pair the query with the task instruction and format it with the model's native processor and chat template.
Encode queries and candidates independently.
Extract the final-layer hidden state at the last non-padding token.
L2-normalize the query and candidate embeddings.
Rank candidates by cosine similarity, equivalently the dot product of the normalized vectors.

No answer generation or query-candidate cross-attention is used during retrieval. Classification labels, passages, images, documents, videos, audio clips, and multimodal items are all treated as candidates in the same embedding space.

Performance

MMEB-v3

MMEB-v3 is an omni-modal benchmark comprising 190 datasets across image, video, visual-document, text, audio, and agent retrieval. Ovis-Omni-Embedding-3B achieves 58.46 overall, outperforming the strongest compared baseline by 5.19 points, and ranks first on the aggregate score of every modality group.

Group	Ovis-Omni-Embedding-3B	Best compared baseline	Margin
Image	77.55	73.83	+3.72
Video	64.99	59.37	+5.62
Visual document	78.26	75.37	+2.89
Text	47.15	43.62	+3.53
Audio	50.08	43.17	+6.91
Agent	45.52	39.42	+6.10
All 190 datasets	58.46	53.27	+5.19

Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Omni-Embedding-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.

Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.

Additional benchmark results

Benchmark	Ovis-Embedding-Omni-3B	Best compared baseline	Evaluation scope
MAEB (beta)	57.29	LCO-Embedding-Omni-7B: 53.54	Mean over 30 audio embedding tasks
MVEB (beta)	61.77	LCO-Embedding-Omni-7B: 57.58	Mean over 23 video and audio-video embedding tasks
RTEB	67.35	Qwen3-Embedding-4B: 67.27	15-task English public retrieval split

These benchmark families use their own official aggregation procedures, so their scores should not be averaged together. MAEB and MVEB results are local evaluations inserted into the corresponding leaderboard snapshots, as described in the technical report.

Embedding dimensions

The native output width is 2048 because no embedding projection head is added to the backbone. For deployments with tighter storage or latency budgets, the post-hoc elastic-dimension module supports 1024, 512, 256, and 128 dimensions. It combines a shared, modality-balanced PCA rotation with a zero-initialized residual linear adapter and folds both operations into one projection matrix at inference time.

Always use the same dimensionality and transformation for queries and candidates, and L2-normalize after projection.

Intended use

The model is intended for embedding extraction and retrieval over supported unimodal or interleaved multimodal content, including:

semantic and cross-modal search;
multimodal RAG indexing and retrieval;
image and visual-document retrieval;
video and audio retrieval;
recommendation and nearest-neighbor matching;
retrieval of tools, GUI states, and memory for agents.

Limitations

This checkpoint produces retrieval embeddings; it is not intended as a speech or text generation model.
Retrieval quality depends on using task-appropriate query instructions and the model's native preprocessing and chat template.
Performance varies by task. In the reported MMEB-v3 comparison, MultiConIR is the principal weakness, while several established visual-document suites and memory retrieval remain below the best specialist result.
Benchmark scores may not directly predict performance on a new domain. Evaluate with representative queries, candidates, and retrieval metrics before deployment.
For high-stakes applications, embeddings should be combined with domain-specific evaluation, access controls, and, where appropriate, a second-stage reranker.

License

This model is released under the Apache 2.0 license.

Resources

This repository contains the weights for Ovis-Omni-Embedding-3B.

Ovis-Omni-Embedding-3B — BitsAndBytes 8-bit derivative

This repository is a quantized derivative of the original model:

Original model: ATH-MaaS/Ovis-Omni-Embedding-3B
Source revision used: 926092ea607bdba164c9985a22a2b6130d8b6118
Quantization: BitsAndBytes 8-bit (load_in_8bit=True)
Native hidden width: 2048
Validated output width: 1024

See QUANTIZATION.md for conversion and validation details.

Task: Feature Extraction
Library: transformers
Type: qwen2_5_omni
License: apache-2.0
Created: Sep 23, 2026
Updated: Sep 23, 2026