jina-embeddings-v3-hf

<br><br>

<p align="center"> <img src="https://huggingface.co/datasets/jinaai/documentation-images/resolve/main/logo.webp" alt="Jina AI: Your Search Foundation, Supercharged!" width="150px"> </p> <p align="center"> <b>The embedding model trained by <a href="https://jina.ai/"><b>Jina AI</b></a>.</b> </p> <p align="center"> <b>jina-embeddings-v3: Multilingual Embeddings With Task LoRA</b> </p>

Native Transformers Implementation

This model uses a native Transformers implementation supported by transformers >= 5.4. It does not require trust_remote_code=True and works out of the box with AutoModel and AutoTokenizer. Task-specific LoRA adapters are loaded via the standard Transformers adapter API.

If you are using an older version of Transformers, please use the original implementation which relies on trust_remote_code=True.

Quick Start

Blog | Azure | AWS SageMaker | API

Using Transformers

The base model can be loaded directly with AutoModel. To use a task-specific LoRA adapter, load it from the corresponding subfolder in the repository and activate it with set_adapter:

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("jinaai/jina-embeddings-v3-hf", dtype=torch.float32)

# Load and activate the retrieval_query LoRA adapter
# Available tasks: retrieval_query, retrieval_passage, separation, classification, text_matching
task = "retrieval_query"
model.load_adapter("jinaai/jina-embeddings-v3-hf", adapter_name=task, adapter_kwargs={"subfolder": task})
model.set_adapter(task)
model.eval()

tokenizer = AutoTokenizer.from_pretrained("jinaai/jina-embeddings-v3-hf")
texts = ["How is the weather today?", "What is the current weather like today?"]
encoded = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
    outputs = model(**encoded)

# Mean pooling
attention_mask = encoded["attention_mask"].unsqueeze(-1)
embeddings = (outputs.last_hidden_state * attention_mask).sum(dim=1) / attention_mask.sum(dim=1)
embeddings = F.normalize(embeddings, p=2, dim=1)

print(embeddings.shape)
# Cosine similarity
print(embeddings @ embeddings.T)

Using Sentence Transformers

from sentence_transformers import SentenceTransformer

# Load model with a task-specific adapter via config_kwargs
task = "retrieval_query"  # retrieval_passage, separation, classification, text_matching
model = SentenceTransformer("jinaai/jina-embeddings-v3-hf", config_kwargs={"subfolder": task})

sentences = ["How is the weather today?", "What is the current weather like today?"]
embeddings = model.encode(sentences)

print(embeddings.shape)

Intended Usage & Model Info

jina-embeddings-v3 is a multilingual multi-task text embedding model designed for a variety of NLP applications. Based on the Jina-XLM-RoBERTa architecture, this model supports Rotary Position Embeddings to handle long input sequences up to 8192 tokens. Additionally, it features 5 LoRA adapters to generate task-specific embeddings efficiently.

Key Features:

• Extended Sequence Length: Supports up to 8192 tokens with RoPE.
• Task-Specific Embedding: Load task-specific LoRA adapters from the corresponding subfolder:
  • retrieval_query: Used for query embeddings in asymmetric retrieval tasks
  • retrieval_passage: Used for passage embeddings in asymmetric retrieval tasks
  • separation: Used for embeddings in clustering and re-ranking applications
  • classification: Used for embeddings in classification tasks
  • text_matching: Used for embeddings in tasks that quantify similarity between two texts, such as STS or symmetric retrieval tasks
• Matryoshka Embeddings: Supports flexible embedding sizes (32, 64, 128, 256, 512, 768, 1024), allowing for truncating embeddings to fit your application.

Supported Languages:

While the foundation model supports 100 languages, we've focused our tuning efforts on the following 30 languages: Arabic, Bengali, Chinese, Danish, Dutch, English, Finnish, French, Georgian, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Thai, Turkish, Ukrainian, Urdu, and Vietnamese.

Contact

Join our Discord community and chat with other community members about ideas.

License

jina-embeddings-v3 is listed on AWS & Azure. If you need to use it beyond those platforms or on-premises within your company, note that the models is licensed under CC BY-NC 4.0. For commercial usage inquiries, feel free to contact us.

Citation

If you find jina-embeddings-v3 useful in your research, please cite the following paper:

@misc{sturua2024jinaembeddingsv3multilingualembeddingstask,
      title={jina-embeddings-v3: Multilingual Embeddings With Task LoRA}, 
      author={Saba Sturua and Isabelle Mohr and Mohammad Kalim Akram and Michael Günther and Bo Wang and Markus Krimmel and Feng Wang and Georgios Mastrapas and Andreas Koukounas and Andreas Koukounas and Nan Wang and Han Xiao},
      year={2024},
      eprint={2409.10173},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.10173}, 
}

Acknowledgments

Thanks to Sai Suraj for providing the Transformers-native implementation.