fincse-multilingual-xlm-roberta-base

🍊 FinCSE-Multilingual

1. INTRO

• Multilingual Financial SimCSE for searching financial/economical sentences.
• Fine-tuned on financial sentence pairs in six languages (KO, EN, ZN, JA, VI, ID), while remaining functional for other pre-trained languages.
• Reference : SimCSE Paper, Original Code

2. Model

• The FinCSEs are built upon multilingual PLMs, such as mBERT and XLM-RoBERTa.
• The FinCSE code in this repository is not derived from the original SimCSE code but is newly implemented. [code]

3. Training Data

• The FinCSEs are fine-tuned using the 'Financial Domain Multilingual Parallel Corpus (금융 분야 다국어 병렬 말뭉치 데이터)' provided by AI HUB in South Korea.
• In this corpus, approximately 2.5 million English, Chinese, Japanese, Vietnamese, and Indonesian sentences in the financial domain are paired with their corresponding Korean sentences.

(KO) 이처럼 금융상품의 경우 판매단계에서 금융회사의 ... 상품의 권유는 기본이고 필수라 할 것이다.
(ZN) 像这样，金融商品在销售阶段，提供金融公司适当的信息和推荐适合金融消费者的商品是基本，也是必须的.

• Since this corpus doesn't have similarity scores like a general STS dataset, performance evaluation is based on a sentence retrieval task, which searches for corresponding sentences in other languages.

(※ Note that this corpus is authorized only for individuals with South Korean citizenship.)

4. Checkpoints

• You can use FinCSE models based on mBERT and XLM-RoBERTa by downloading them from following links.

  • fincse-multilingual-bert-cased
  • fincse-multilingual-xlm-roberta-base
  • fincse-multilingual-xlm-roberta-large

• The performance of these FinCSE models on financial sentence retrieval task is as follows. (This performance is evaluated using 50,000 samples from the dataset, which were not used in the fine-tuning.)

(※ It is recommended to use the FinCSE model based on XLM-RoBERTa, as the mBERT-based model is unstable.)

5. Examples

• Query : 중국 생성형 AI 기업 현황 (= Status of generative AI companies in China)
• Texts :

import numpy as np
from transformers import AutoModel, AutoTokenizer

model_path = 'snumin44/fincse-multilingual-xlm-roberta-base'
model = AutoModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

query = '중국 생성형 AI 기업 현황'

targets = [
    "Experts predict that the U.S. Federal Reserve will lower interest rates by 0.25 percentage points this month, predicting that interest rates will fall by more than 0.5 percentage points in total within this year.",
    "铁矿石是高度依赖中国需求的原材料，中国占各矿业公司出口的70%。",
    "オラクルは今年の大型技術株のうち、株価上昇率が人工知能（ＡＩ）半導体大将主であるＮＶＩＤＩＡの１３９％上昇率を除けば最も高い。",
    "NYT chỉ ra rằng điểm yếu của ngành công nghiệp Trung Quốc là LLM và hầu hết các chương trình mà các công ty Trung Quốc đưa ra dưới dạng AI tổng hợp trên thực tế đều được nhập khẩu từ Mỹ và được cải tiến.",
    "Harga saham Trump Media, perusahaan induk Truth Social, perusahaan layanan jejaring sosial (SNS) yang didirikan oleh mantan calon presiden AS dari Partai Republik Donald Trump, anjlok lebih dari 10%."
]

query_feature = tokenizer(query, return_tensors='pt')
query_outputs = model(**query_feature, return_dict=True)
query_embeddings = query_outputs.pooler_output.detach().numpy().squeeze()

def cos_sim(A, B):
    return np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B))

for idx, target in enumerate(targets):
    target_feature = tokenizer(target, return_tensors='pt')
    target_outputs = model(**target_feature, return_dict=True)
    target_embeddings = target_outputs.pooler_output.detach().numpy().squeeze()
    similarity = cos_sim(query_embeddings, target_embeddings)
    print(f"Similarity between query and target {idx}: {similarity:.4f}")

Similarity between query and target 0: 0.1105
Similarity between query and target 1: 0.2662
Similarity between query and target 2: 0.3043
Similarity between query and target 3: 0.5328
Similarity between query and target 4: 0.1904

Citing

@article{gao2021simcse,
   title={{SimCSE}: Simple Contrastive Learning of Sentence Embeddings},
   author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
   booktitle={Empirical Methods in Natural Language Processing (EMNLP)},
   year={2021}
}