Downloads · 30 days
8
15% of all-time downloads
gubernac/jina-embeddings-v5-omni-nano-ONNX
jina-embeddings-v5-omni-nano-ONNX is a feature extraction model from gubernac. Use it when you need embeddings to search or compare text. It is set up for transformers.js. The card lists the license as cc-by-nc-4.0.
ONNX exports of jinaai/jina-embeddings-v5-omni-nano for in-browser inference via transformers.js v4 with WebGPU.
Downloads · 30 days
8
15% of all-time downloads
All-time downloads
55
Public
Repo size
10.4 GB
Likes
1
Public
Click a slice to open those files.
.data5.5 GB · 52%
From the Hugging Face model README
ONNX exports of jinaai/jina-embeddings-v5-omni-nano for in-browser inference via transformers.js v4 with WebGPU.
jina-embeddings-v5-omni-nano is a ~1.04 B-parameter multimodal embedding model that maps text, images, audio, and video into a single shared 768-dimensional L2-normalized space, enabling cross-modal retrieval without reindexing.
The base model is a LLaVA-style composition of three frozen encoders plus small trainable projectors, with per-task LoRA adapters. This repo's exports have the retrieval task adapter merged into the static weights.
| Tower | Backbone | Role |
|---|---|---|
| Text | EuroBERT-210m (loaded as a bidirectional Llama) | Text encoder |
| Vision | Qwen3-VL vision tower + spatial merger | Image and video frames |
| Audio | Whisper-large-v3 encoder + Qwen2.5-Omni audio adapter | Audio (16 kHz mono) |
Each tower is bundled with its retrieval projector AND a copy of the language model (which performs the final cross-modal fusion + last-token pooling), so the three ONNX graphs are independently loadable.
Three ONNX graphs, three quantization variants each. Every variant ships as a small .onnx schema + a large .onnx.data or .onnx_data external-weight sidecar (HF Hub-friendly layout, no 2 GB protobuf limits).
| Modality | fp32 | fp16 | q4f16 | Verified parity vs fp32 (fp16) |
|---|---|---|---|---|
| Text | 849 MB | 424 MB | 263 MB | cos = 1.000000 |
| Vision | 1247 MB | 622 MB | 460 MB | cos = 0.999998 |
| Audio | 3400 MB | 1700 MB | 1465 MB | (fp16 doesn't load on CPU EP; verify on WebGPU) |
For desktop WebGPU, use q4f16 — it's the smallest and runs natively on shader-int4 hardware. Use fp16 if you need higher numerical fidelity or your GPU lacks int4 paths.
Every graph outputs a single tensor sentence_embedding of shape [batch, 768], already L2-normalized — cosine similarity reduces to a dot product.
Text (text_model*.onnx): inputs input_ids and attention_mask, both [batch, seq] with seq fully dynamic. Apply the asymmetric retrieval prefix convention before tokenizing: Query: … for queries, Document: … for corpus items.
Vision (vision_model*.onnx): the graph is traced at a fixed image-patch layout. image_grid_thw is folded as a constant (it drives torch.linspace inside Qwen3-VL's fast_pos_embed_interpolate, which dynamo cannot symbolicate). Resize every image to 224×224 before passing through LlavaEuroBertProcessor — that yields the exact shapes the graph expects:
input_ids [batch, 271] int64
attention_mask [batch, 271] int64
pixel_values [1024, 1536] float32
image_grid_thw is NOT an ONNX input (it's a constant). Don't pass it.
Audio (audio_model*.onnx): traced with a 5-second 16 kHz mono clip. The graph expects exactly:
input_ids [batch, 125] int64 (audio_token_id placeholders)
attention_mask [batch, 125] int64
input_features [1, 128, 3000] float32 (Whisper log-mel, 30s padded)
feature_attention_mask [1, 3000] int64 (frame-level, 500 ones for 5s of real audio)
Pad or truncate every clip to 5 s. Longer-clip chunking (sliding 5 s windows, average pooled embeddings) is a v2 follow-up.
import { AutoTokenizer } from "@huggingface/transformers";
import * as ort from "onnxruntime-web";
const REPO = "shreyask/jina-embeddings-v5-omni-nano-ONNX";
const tok = await AutoTokenizer.from_pretrained(REPO);
const sess = await ort.InferenceSession.create(
`https://huggingface.co/${REPO}/resolve/main/onnx/text_model_q4f16.onnx`,
{ executionProviders: ["webgpu", "wasm"] },
);
const { input_ids, attention_mask } = await tok(
"Query: a saxophone solo",
{ return_tensors: "ort" },
);
const { sentence_embedding } = await sess.run({ input_ids, attention_mask });
// Float32Array of length 768, already L2-normalized.
import { AutoProcessor } from "@huggingface/transformers";
import * as ort from "onnxruntime-web";
const proc = await AutoProcessor.from_pretrained(REPO);
const sess = await ort.InferenceSession.create(
`https://huggingface.co/${REPO}/resolve/main/onnx/vision_model_q4f16.onnx`,
{ executionProviders: ["webgpu", "wasm"] },
);
// Resize to 224x224 before passing in.
const inputs = await proc.apply_chat_template(
[{ role: "user", content: [{ type: "image", image: imageBlob }] }],
{ add_generation_prompt: false, tokenize: true, return_dict: true, return_tensors: "ort" },
);
const { sentence_embedding } = await sess.run({
input_ids: inputs.input_ids,
attention_mask: inputs.attention_mask,
pixel_values: inputs.pixel_values,
});
Audio preprocessing isn't bundled in LlavaEuroBertProcessor — use Whisper's feature extractor directly and stamp audio_token_id placeholders into input_ids. See the reference implementation for the exact mel-spec → placeholder count plumbing.
All three towers project into the same 768-dim space, so a text query can rank images / audio / video corpus items (and vice versa) without re-indexing. Embeddings are L2-normalized, so cosine similarity is a dot product:
const score = textVec.reduce((s, v, i) => s + v * imageVec[i], 0);
torch 2.11, transformers 5.8, onnx 1.21, onnxruntime 1.26torch.onnx.export(..., dynamo=True). The LlamaModel-based encoder exports cleanly through the dynamo pathtorch.onnx.export(..., dynamo=False) (legacy TorchScript tracer). Dynamo refuses to specialize Qwen3-VL's data-dependent torch.linspace, and the GQA-aware SDPA is monkey-patched with a manual MatMul+Softmax for the trace's durationmerge_and_unload(safe_merge=True) so the retrieval task adapter is baked inonnxruntime.transformers.float16.convert_float_to_float16 (the onnxconverter_common path mishandles dynamo's _to_copy nodes)onnxruntime.quantization.matmul_nbits_quantizer.MatMulNBitsQuantizer (bits=4, block_size=32, accuracy_level=4)Cast(to=int64) inserted before every Slice index input (366 inserts), and Unsqueeze(0)/Squeeze(0) wrapped around the rank-2-input AveragePool (1 site)Inherited from the base model: CC BY-NC 4.0. Commercial use requires reaching out to [email protected].
@misc{jina-embeddings-v5-omni-nano,
author = {Jina AI},
title = {jina-embeddings-v5-omni-nano},
year = {2025},
url = {https://huggingface.co/jinaai/jina-embeddings-v5-omni-nano}
}