Conan-embedding-v2

Conan-Embedding-v2

What's New?

• Performance

  Conan-Embedding-v2 has now achieved SOTA performance on the MTEB leaderboard for both Chinese and English.

• Cross-lingual Retrieval between Chinese and English

  Conan-Embedding-v2 supports cross-lingual retrieval between Chinese and English samples.

• Longer Context Support

  Conan-Embedding-v2 now supports a context length of 32,768 tokens.

• Conan 1.4B Large Model Trained from Scratch

  A vocabulary and large language model trained from scratch, with a pre-trained model and vocabulary more tailored to the Embedding scenario, delivering stronger performance.

  The Conan-1.4B base model will be open-sourced. Community workers can train their own Embedding models based on the Conan-1.4B base model.

Performance

Performance of Conan-Embedding-v2 on MTEB for Chinese and English

English

Chinese

Model Detail

Model Structure

Conan-Embedding-v2 Structure:

SentenceTransformer(  
    (0): Transformer({
        'max_seq_length': 32768, 
        'do_lower_case': False
        }) with Transformer model: ConanEmbedModel,
    (1): Pooling({
        'word_embedding_dimension': 3584, 
        'pooling_mode_cls_token': False, 
        'pooling_mode_mean_tokens': True, 
        'pooling_mode_max_tokens': False, 
        'pooling_mode_mean_sqrt_len_tokens': False, 
        'pooling_mode_weightedmean_tokens': False, 
        'pooling_mode_lasttoken': False, 
        'include_prompt': True
        }),
    (2): Dense({
        'in_features': 3584, 
        'out_features': 3584, 
        'bias': True, 
        'activation_function': 'torch.nn.modules.linear.Identity'
        })
)

Key Specifications of Conan-1.4B (Transformer):

• Number of Parameters (Non-Dense-Layer): 1.48B

• Vocabulary Size: 150,000

• Number of Layers: 8

• Hidden Layer Dimension: 3584

• Number of Attention Heads (GOA): 32 for Q and 8 for KV

• Intermediate Dimension of FFN Layer: 8192

• Maximum Context Window: 32,768 Tokens

For more model details, please refer to model/modeling_conan.py and config.json, or stay tuned for the upcoming open-source release of Conan-1.4B Base Model.

Tokenizer

We trained the Tokenizer on a large-scale multilingual dataset to build a standard BBPE(Byte-level Byte Pair Encoding) tokenizer with a vocabulary size of 150,000.

Technical Report

We will soon release our technical report.

Using Conan-Embedding-v2

Use /model/conan_api_client.py to access our test API. A sample call is as follows:

from modeling_conan import ConanClient

AK = os.getenv("CONAN_AK")
SK = os.getenv("CONAN_SK")
client = ConanClient(ak=AK, sk=SK, url="https://ai.om.qq.com/api/conan/v2")
res = client.embed("Hello!")
print(res)

This is a temporary calling solution. Please contact us to obtain an access token.

In the future, we will provide high-performance, cost-effective, and reliable Embedding services on Tencent Cloud.

---

About

Created by the Tencent BAC Group. All rights reserved.