prominer-ru-pretrained-cross-encoder

ProMiNER Russian BioNNE-L Dictionary-Pretrained Cross-Encoder

Cross-encoder pretrained on UMLS dictionary pseudo-pairs with compact candidate-context profiles.

This model is part of ProMiNER, a Russian-track biomedical entity-linking system for BioNNE-L. The system links mentions from NEREL-BIO/BioNNE-L texts to UMLS concepts by combining dense retrieval and cross-encoder reranking.

• GitHub: https://github.com/bikingSolo/prominer
• Model type: sentence-transformers-cross-encoder
• Base/initializer: andorei/BERGAMOT-multilingual-GAT

Training

Pretrained from dictionary-derived pseudo-query/candidate pairs. Candidate concepts are represented by compact profiles containing representative names and selected aliases. This checkpoint is intended as the initializer for task-specific BioNNE-L reranking.

Hyperparameters:

• reranker_model_name_or_path: andorei/BERGAMOT-multilingual-GAT
• retriever_model_name_or_path: andorei/BERGAMOT-multilingual-GAT
• loss_name: bce
• epochs: 5
• train_batch_size: 128
• learning_rate: 2e-05
• weight_decay: 0.01
• warmup_ratio: 0.1
• max_seq_length: 384
• pretrain_num_negatives: 20
• max_pseudo_queries_per_cui: 5
• num_train_pairs: 5900264
• num_eval_pairs: 317280
• selection_metric: Acc@1

According to the Acc@1 on dev, the best epoch is 2.

Full local metadata exported from MLflow is included in prominer_metadata/.

Evaluation

Metrics below are copied from the local MLflow run artifacts.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("bikingSolo/prominer-ru-pretrained-cross-encoder", num_labels=1)
scores = model.predict([
    (
        "вестибулокохлеарный нерв",
        "слуховой нерв; вестибулокохлеарный нерв; nervus vestibulocochlearis [viii]",
    )
])

Intended Use

This checkpoint is intended for research and reproducibility of the ProMiNER BioNNE-L Russian entity-linking pipeline. For the full system, use:

1. prominer-ru-retriever to retrieve candidate UMLS concepts.
2. prominer-ru-reranker to rerank those candidates with candidate-context profiles.

The dictionary-pretrained cross-encoder is primarily an intermediate checkpoint used to initialize the final reranker.

Data and Citation

Training and evaluation use BioNNE-L/NEREL-BIO resources and UMLS-derived terminology available in this repository's data layout. Cite the relevant NEREL-BIO and BioNNE-L papers when using this model.

Check https://github.com/bikingSolo/prominer for more info.