sci-rus-tiny4

mlsa-iai-msu-lab/sci-rus-tiny4

Model Description

This is a multilingual encoder model designed for scientific text embeddings, supporting Russian, English, and Chinese languages. It is built on the ModernBERT architecture and trained on a large corpus of scientific papers, citations, and co-citations to capture semantic similarity across these languages.

Training Stages

The model was trained from scratch in two stages: Masked Language Modeling (MLM), followed by a joint Contrastive Learning and Knowledge Distillation stage. A custom tokenizer was also trained from scratch to better handle scientific terminology across the three languages.

1. Masked Language Modeling (MLM)

In the first stage, the model was trained from scratch using the MLM objective on 31 million scientific abstracts collected from Elibrary, Semantic Scholar (S2), and ScienceChina.

2. Contrastive Learning + Knowledge Distillation (Joint)

In the second stage, contrastive learning and knowledge distillation are optimized together in a single training run (launched by a distillation training script). The total loss is a weighted combination of an InfoNCE contrastive loss and a KL-divergence distillation loss between the student and teacher pairwise-similarity distributions:

$$\mathcal{L} = (1 - w) \cdot \mathcal{L}{\text{InfoNCE}} + w \cdot \mathcal{L}{\text{KL}}, \quad w = 0.7$$

The student is distilled from the teacher Qwen/Qwen3-Embedding-8B to transfer its representation quality into the compact ModernBERT student. Teacher embeddings are used in offline mode — precomputed per-sample embeddings looked up at training time rather than running the teacher online — with a distillation (softmax) temperature of 0.1.

Training Data & Sampling (Contrastive Stage)

The model is trained on data from three primary sources:

• Elibrary (Ru-En)
• Semantic Scholar (S2) (En)
• ScienceChina (Zh-En)

Two types of datasets were used during training:

1. Title - Abstract Pairs:

   • Pairs consist of the title and abstract of the same paper.
   • Cross-lingual sampling: Upsampling (8x) is applied to cross-lingual pairs (e.g., Russian Title - English Abstract) to improve multilingual vector space alignment.

2. Citation and Co-citation Pairs:

   • Pairs consist of two related papers (e.g., Article A cites Article B).
   • Text Selection: For each article, either the title or abstract is randomly selected.
   • Cross-lingual sampling: There is a 50% probability of selecting texts in different languages (if available) to reinforce multilinguality.

Dataset Statistics