multivector-gte-modernbert-base-miriad

ColBERT gte-modernbert-base trained on MIRIAD question-passage pairs

This is a Multi-Vector Encoder model trained on the miriad-4.4_m-split dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modality: Text
• Training Dataset:
  • miriad-4.4_m-split
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-gte-modernbert-base-miriad")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'How does treatment with CZP plus MTX impact disease activity in patients with rheumatoid arthritis?\n',
]
documents = [
    "ACR20 response rates were significantly higher with CZP plus MTX than placebo plus MTX at Week 1 (22.9 and 14.3% with CZP 200 mg plus MTX vs 5.6 and 3.3% with placebo plus MTX in the RAPID 1 and 2 trials, respectively) [4, 5] . ACR20 response rates peaked at Week 12 in both studies (63.8 and 62.7% for CZP 200 mg vs 18.3 and 12.7% for placebo in RAPID 1 and 2, respectively; both P < 0.001). At Week 24, ACR20 response rates were 58.8 and 57.3% for patients receiving CZP 200 mg plus MTX, respectively, vs 13.6 and 8.7%. The ITT populations for RAPID 1 and 2 consisted of all patients who were randomized into the studies; the modified ITT population for FAST4WARD consisted of all randomized patients who had taken one or more dose of study medication. Adapted from Mease [21] with permission of Future Medicine Ltd. CV: coefficient of variation; ITT: intention-to-treat; NA: not applicable.\n\n Significantly higher ACR50 and ACR70 response rates for CZP vs placebo groups were seen from Weeks 2 and 4 in RAPID 1, and Weeks 6 and 20 in RAPID 2, respectively. Responses were sustained to the end of the trials (Week 52 in RAPID 1 and Week 24 in RAPID 2; Table 2 ), and were similar in the CZP 400 mg plus MTX groups. CZP treatment also yielded significant improvements in all ACR core component scores, including reductions in swollen and tender joint scores and improvements in both patient's and physician's global assessments of disease activity, by Week 1 that were sustained throughout both studies [4, 5] . Treatment with CZP plus MTX was associated with significantly greater improvements in disease activity from Week 1, as evidenced by DAS-28 (ESR) scores, throughout both trials (P < 0.001 at all time points) [4, 5] . At Week 1, mean change from baseline in DAS-28 was À0.8 with CZP 200 mg and À0.3 with placebo in RAPID 1, and À0.8 with CZP 200 mg and À0.2 with placebo in RAPID 2. Improvements were sustained to the end of both studies (52 or 24 weeks, respectively; Fig. 1 ), and were similar with the CZP 400 mg dose. In RAPID 2, DAS-28 remission was observed in 9.4% of patients treated with CZP 200 mg plus MTX compared with only 0.8% of patients in the placebo group [5] .\n\n Both trials investigated the effects of CZP on the progression of joint damage. In RAPID 1, the mean (S.D.) change in mTSS from baseline to Week 52, which was a co-primary endpoint of the study, was significantly lower in patients receiving CZP 200 mg plus MTX [0.4 (5.7) in the CZP 200 mg group] compared with patients receiving placebo plus MTX [2.8 (7.8); P < 0.001] [4] . The changes were also significantly lower in the CZP plus MTX groups vs the placebo plus MTX group at Week 24 (P < 0.001). At both time points, significantly lower mean changes from baseline in both erosion (Week 24: 0 vs 0.7, Week 52: 0.1 vs 1.5; P < 0. [5] . Patients in the CZP 200 mg group in RAPID 2 also had significantly lower erosion (mean change from baseline: 0.1 vs 0.7) and joint space narrowing (mean change from baseline: 0.1 vs 0.5) subscores (P 4 0.01). Results for patients receiving the 400-mg dose were similar. An analysis of joint damage in patients who withdrew from the trials at Week 16 due to ACR20 non-response at Weeks 12 and 14 (as mandated by the study protocol) found that radiographic progression was inhibited by CZP plus MTX despite the fact that these patients did not meet the threshold for a clinical response [4, 5] .",
    'Five minutes after atropine, the R:T ratio increased from 1.15 (0.4) to 1.40 (0.6) (P < 0.01); at 30 min it was 1.51 (0.7) (P < 0.001) and at 60 min it was 1.33 (0.5) (P < 0.05). The R-wave amplitude was not affected by atropine. No changes in heart rate, QTc interval, RSA and R:T ratio occurred after placebo. COMMENT These data show that, in the presence of vagal block by atropine, the QTc interval increased significantly and the T-wave of the ECG was flattened.\n\n We chose a relatively large dose of atropine to ensure parasympathetic block as confirmed by the disappearance of RSA. Day, McComp and Campbell fl] have suggested that QT dispersion (interlead variability) gives an indication of arrythmogenicity and repolarization. We used a single lead V 2 which, according to the same group, provides the closest approximation to maximum QT interval [4] . They also accept the validity of a single lead value for QTc when changes are monitored. The flattened T-wave after atropine in our volunteers probably also reflected irregularity in repolarization.\n\n Atropine has been shown to increase the incidence of cardiac arrhythmia during induction of anaesthesia [3] . In addition, i.v. atropine has been shown to cause ventricular tachycardia in a patient with a prolonged QT interval syndrome [5] . Inhibition of the sympathoadrenal tone by opioids shortens the QTc interval in patients with vagal block. Vagal stimulation protects the heart against arrhythmogenic vulnerability [2] and against prolongation of the QT interval. In our study, the QTc interval was prolonged, probably because sympathoadrenal tone became dominant after parasympathetic block by atropine.\n\n In diabetic patients, vagal denervation develops gradually. Maintenance of remaining borderline vagal function by avoiding anticholinergics may be of value in diabetic patients, as serious cardiac arrhythmia has been described in these patients during anaesthesia and after atropine. Furthermore, ventricular fibrillation after i.v. atropine for bradycardia has been shown to occur in acute myocardial infarction [6] . The routine use of anticholinergics at induction of anaesthesia must be seriously questioned.',
    'This occurs since, in the folded state, the dansyl group is encapsulated in the hydrophobic cavity of the β-cyclodextrin ring resulting in a net fluorescence enhancement [99] . As a further development of this work, Riccardi and co-workers have described a tris-conjugated TBA 15 (tris-mTBA), equipped with a dansyl, a β-cyclodextrin and a biotin tag at the ends. This novel design has allowed the incorporation of TBA 15 onto streptavidin-coated NPs, leading to a remarkable increase of its anticoagulant properties. The developed systems have provided the basis for suitable aptamer-based devices for theranostic applications, allowing simultaneously both fluorescence-based detection and modulation of the thrombin activity [101] .\n\n Notably, in addition to the sensing approaches based on conformational switch random coil-G-quadruplex structure, also thrombin-induced changes starting from a hairpin structure are possible if the aptamer is properly engineered. In this context, Hamaguchi et al. have described a TBA 15 elongated at the 5 -end with few nucleotides complementary to the 3 -end and therefore able to adopt a stem-loop structure or hairpin [102] . In addition, the aptamer is equipped with a fluorescent/quencher pair, i.e., a fluorescein and a dabcyl moiety at the 5 -and 3 -end, respectively. In the absence of thrombin, the close proximity between the two reporter groups in the hairpin structure determines fluorescence quenching. After thrombin recognition, the stem-loop structure is destabilized in favour of interactions with the protein. Under these conditions, the fluorescent dye and the quencher are distant, thus allowing a "turn-on" of the fluorescence signal, indicative of the binding with the target molecule ( Figure 7c ).\n\n Alternative approaches for "structure switch signalling aptamers" are reported by Nutiu and Li [103] . Their strategy for designing aptamer-based fluorescent reporters involves structural switches from DNA/DNA duplex to DNA/target complex. In this study, the aptamer beacon consists of a tripartite duplex structure including a 5 -fluorescein-labeled oligomer (FDNA), a 3 -dabcyl-labeled oligomer (QDNA) and a longer oligonucleotide sequence comprising Stem-1 and Stem-2, complementary to FDNA and QDNA, respectively. Stem-2 also contains the TBA 15 sequence in a partial overhang (Figure 8a ). In the absence of the target protein, the aptamer naturally binds to FDNA and QDNA, bringing the fluorophore and the quencher in close proximity and thus completely inhibiting the fluorescence signal. The presence of thrombin triggers the formation of the aptamer-target complex, causing the release of QDNA and fully restoring the fluorescence emission.\n\n Cancers 2017, 9, 174 10 of 43\n\n Notably, in addition to the sensing approaches based on conformational switch random coil-G-quadruplex structure, also thrombin-induced changes starting from a hairpin structure are possible if the aptamer is properly engineered. In this context, Hamaguchi et al. have described a TBA15 elongated at the 5′-end with few nucleotides complementary to the 3′-end and therefore able to adopt a stem-loop structure or hairpin [102] . In addition, the aptamer is equipped with a fluorescent/quencher pair, i.e., a fluorescein and a dabcyl moiety at the 5′-and 3′-end, respectively. In the absence of thrombin, the close proximity between the two reporter groups in the hairpin structure determines fluorescence quenching. After thrombin recognition, the stem-loop structure is destabilized in favour of interactions with the protein. Under these conditions, the fluorescent dye and the quencher are distant, thus allowing a "turn-on" of the fluorescence signal, indicative of the binding with the target molecule ( Figure 7c ).\n\n Alternative approaches for "structure switch signalling aptamers" are reported by Nutiu and Li [103] . Their strategy for designing aptamer-based fluorescent reporters involves structural switches from DNA/DNA duplex to DNA/target complex. In this study, the aptamer beacon consists of a tripartite duplex structure including a 5′-fluorescein-labeled oligomer (FDNA), a 3′-dabcyl-labeled oligomer (QDNA) and a longer oligonucleotide sequence comprising Stem-1 and Stem-2, complementary to FDNA and QDNA, respectively. Stem-2 also contains the TBA15 sequence in a partial overhang (Figure 8a ).',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (32, 128) (814, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[26.4624,  9.1034,  3.9515]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Multi Vector Information Retrieval

• Dataset: miriad_eval
• Evaluated with <code>MultiVectorInformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

miriad-4.4_m-split

• Dataset: miriad-4.4_m-split
• Size: 640,000 training samples
• Columns: <code>question</code> and <code>passage_text</code>
• Approximate statistics based on the first 100 samples:

  	question	passage_text
  type	string	string
  modality	text	text
  details	<ul><li>min: 10 tokens</li><li>mean: 22.31 tokens</li><li>max: 60 tokens</li></ul>	<ul><li>min: 541 tokens</li><li>mean: 940.66 tokens</li><li>max: 1415 tokens</li></ul>

• Samples:

  question	passage_text
  <code>What factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction?<br></code>	<code>I n 1966, Ross and Somerville 1 reported the first use of an aortic homograft to establish right ventricle-to-pulmonary artery continuity in a patient with tetralogy of Fallot and pulmonary atresia. Since that time, pulmonary position homografts have been used in a variety of right-sided congenital heart lesions. Actuarial 5-year homograft survivals for cryopreserved homografts are reported to range between 55% and 94%, with the shortest durability noted in patients less than 2 years of age. 4 Pulmonary position homografts also are used to replace pulmonary autografts explanted to repair left-sided outflow disease (the Ross operation). Several factors may be likely to favor increased pulmonary conduit durability in Ross patients compared with those with right ventricular outflow tract obstruction, including later age at operation (allowing for larger homografts), more normal pulmonary artery architecture, absence of severe right ventricular hypertrophy, and more natural positioning of ...</code>
  <code>How does MCAM expression in hMSC affect the growth and maintenance of hematopoietic progenitors?</code>	<code>After culture in a 3-dimensional hydrogel-based matrix, which constitutes hypoxic conditions, MCAM expression is lost. Concordantly, Tormin et al. demonstrated that MCAM is down-regulated under hypoxic conditions. 10 Furthermore, it was shown by others and our group that oxygen tension causes selective modification of hematopoietic cell and mesenchymal stromal cell interactions in co-culture systems as well as influence HSPC metabolism. [44] [45] [46] Thus, the observed differences between Sharma et al. and our data in HSPC supporting capacity of hMSC are likely due to the different culture conditions used. Further studies are required to clarify the influence of hypoxia in our model system. Altogether these findings provide further evidence for the importance of MCAM in supporting HSPC. Furthermore, previous reports have shown that MCAM is down-regulated in MSC after several passages as well as during aging and differentiation. 19, 47 Interestingly, MCAM overexpression in hMSC enhance...</code>
  <code>What is the relationship between Fanconi anemia and breast and ovarian cancer susceptibility genes?<br></code>	<code>( 31 ) , of which 5% -10 % may be caused by genetic factors ( 32 ) , up to half a million of these patients may be at risk of secondary hereditary neoplasms. The historic observation of twofold to fi vefold increased risks of cancers of the ovary, thyroid, and connective tissue after breast cancer ( 33 ) presaged the later syndromic association of these tumors with inherited mutations of BRCA1, BRCA2, PTEN, and p53 ( 16 ) . By far the largest cumulative risk of a secondary cancer in BRCA mutation carriers is associated with cancer in the contralateral breast, which may reach a risk of 29.5% at 10 years ( 34 ) . The Breast Cancer Linkage Consortium ( 35 , 36 ) also documented threefold to fi vefold increased risks of subsequent cancers of prostate, pancreas, gallbladder, stomach, skin (melanoma), and uterus in BRCA2 mutation carriers and twofold increased risks of prostate and pancreas cancer in BRCA1 mutation carriers; these results are based largely on self-reported family history inf...</code>

• Loss: <code>CachedMultiVectorMultipleNegativesRankingLoss</code> with these parameters:

  {
      "score_metric": "colbert_scores",
      "mini_batch_size": 8,
      "mini_batch_num_tokens": null,
      "score_mini_batch_size": 8,
      "scale": 1.0,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates
• max_length: 1024

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 128
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 3e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• max_length: 1024

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Training Time

• Training: 5.1 hours
• Evaluation: 13.4 minutes
• Total: 5.4 hours

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.22.2

Additional Resources

• Sentence Transformers Documentation: the full documentation site, including training, evaluation, and pre-trained model catalogs.
• PyLate: the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->