bge-m3-fa-CLP-interMoE

Enhanced-BGE-M3-with-CLP-and-MoE (paper, code)

Contrastive Learning Penalty (CLP)

CLP is a novel loss function designed to address the limitations of existing contrastive learning methods for improved performance in information retrieval tasks. It incorporates a penalty term that encourages the model to learn more discriminative representations by considering the similarity between negative samples and their corresponding queries.

The CLP loss function is defined as follows:

<img src="https://raw.githubusercontent.com/CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE/main/imgs/clpl_formula.PNG" width="1000"/>

where:

• h_i: The embedding of the query for the i-th instance.
• h_i⁺: The embedding of the positive sample for the i-th instance.
• H^': The set of negative samples for the i-th instance.
• h^': The embedding of the negative sample's query.
• H^*: the set of positive queries for the documents corresponding to the negative samples
• sim(a, b): The cosine similarity function between embeddings a and b.
• τ: The temperature parameter.
• λ: The balancing parameter between the contrastive loss and the penalty term.

The difference between Contrastive Learning Loss and Contrastive Learning Penalty Loss:

<img src="https://raw.githubusercontent.com/CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE/main/imgs/figure1.PNG" width="1000"/>

Specs

• Model

• Data

Performing negative sampling using the ANCE methodology and generating negative sample's positive queries through the Gemini 1.5 Pro model, which are required for CLPL.

Evaluation

<img src="https://raw.githubusercontent.com/CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE/main/imgs/table4.PNG" width="1000"/>

Citation

@misc{yu2024efficientfinetuningmethodologytext,
  title={Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)}, 
  author={Jeongsu Yu},
  year={2024},
  eprint={2412.17364},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2412.17364}, 
}