ModernBERT-projects-retriever

A retriver version of ModernBERT specifically finetuned for project retrieval. It is build on top of joe32140/ModernBERT-base-msmarco

This is a sentence-transformers model finetuned from joe32140/ModernBERT-base-msmarco. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: joe32140/ModernBERT-base-msmarco <!-- at revision fb03cb72796da2e1a925c8d5aa73e5d8c6b57f5a -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

<!-- - **Training Dataset:** Unknown -->

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'home renovation energy savings platform',
    'Leveraging on the business operations of consortium partners (4000 integrated building renovations conducted to date), TURNKEY RETROFIT will develop and replicate an integrated home renovation service which will be initially operated in 3 EU countries - France, Ireland and Spain - with an expected 335M€ investment pipeline for home renovation within the first 5 years (approximately 14 700 dwellings renovated, leading to 96,6 GWh/year Primary energy savings triggered) beyond the end of the project. The project will point to further replication across Europe and Internationally in particular via the Green Building Council network.The TURNKEY RETROFIT service will be developed as a home-owner-centric renovation journey, which will transform the complex and fragmented renovation process into a simple, straightforward and attractive process for the home-owner. It will include the initial technical and behavioural diagnosis, technical offer, contract development and agreement, structuring and provision of financial support, as well as the on-site coordination of works and quality assurance. It will be a service-oriented model where the home-owner is offered tailor-made solutions through the whole customer journey.The service will be accessible through a user-friendly digital platform and it will address drivers of building renovation that go beyond a desire to reduce energy bills and increase asset value, such as home improvement, increased comfort, enhanced health & quality of life.',
    'BLAZE aims at developing Low cost, Advanced and Zero Emission first-of-a-kind small-to-medium Biomass CHP. This aim is reached by developing bubbling fluidised bed technology integrating high temperature cleaning & conditioning system (IBFBG, that can convert heterogeneous feedstocks in a syngas with zero particulate matter and ultra-low tar and contaminants content), an integrated high temperature gas cleaning approach for HCl and H2S removal and an innovative key component for thermal and chemical integration of solid oxide fuel cell (efficient gas recirculation of the fuel cell anode exhaust to the gasification process via a steam-driven high speed micro-compressor using gas bearing technology). The technology is developed for a CHP capacity range from small (25-100 kWe) to medium (0.1-5 MWe) scale and is characterised by the widest fuel spectrum applicable (forest, agricultural, industrial and municipal waste also with high moisture, ash and contaminants content), high efficiencies (50% electrical versus the actual 20%), low investment (< 4 k€/kWe) and operation (≈ 0.05 €/kWh) costs as well as almost zero gaseous and PM emissions, projecting electricity production cost below 0.10 €/kWh . Gasification, gas cleaning & conditioning and fuel cells will be tested at lab scale and 25 kWe SOFC will be thermally and chemically integrated in 100 kWth IBFBG demonstrating the achievement of new milestones, increasing competitiveness of European industry, energy system reliability and flexibility and biomass plants social acceptance. Process simulations, computer aided design, tests, performance evaluation, risk and safety analysis as well as a technology assessment part covering techno-economic, environmental and overall impact assessments and market studies will be carried out together with a clear dissemination, exploitation and communication plan, that can count on the involvement of the main gasifier, gas conditioning and SOFC European companies and research centres.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.7217,  0.0099],
#         [ 0.7217,  1.0000, -0.0355],
#         [ 0.0099, -0.0355,  1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 25,553 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	<ul><li>min: 6 tokens</li><li>mean: 12.88 tokens</li><li>max: 25 tokens</li></ul>	<ul><li>min: 43 tokens</li><li>mean: 340.97 tokens</li><li>max: 794 tokens</li></ul>

• Samples:

  anchor	positive
  <code>identify neural markers of partner action prediction in joint turn-taking tasks</code>	<code>The ability to perform joint (multi-person) actions such as ensemble music performance confers numerous benefits, from generating unique aesthetic experiences to enhancing affiliation with one’s co-actors. Co-actors must be able to accurately predict when one another’s actions will occur so that they can time their own actions accordingly. One question facing the field of social neuroscience is how co-actors learn to make accurate predictions about one another’s actions. The current project, JAL, launches one of the first systematic investigations of joint action learning to address this question on the levels of both brain and behaviour. The project will implement a series of empirical studies in which partners learn novel joint turn-taking tasks while electroencephalography (EEG) is simultaneously measured from both partners: Turn-taking tasks are optimal for measuring prediction processes because partners must be able to accurately predict when one another’s turns will end so that ...</code>
  <code>measure intergalactic magnetic field strength topology with Jansky VLA observations</code>	<code>There is a pressing need to chart the origin, evolution, and impact of magnetic fields over the full 13.7 billion year history of the Universe, with implications for particle physics, cosmology, and astrophysics. Key to this endeavour is a detailed physical understanding of how galaxies have become magnetised and what role intergalactic magnetic fields play in shaping the Universe. To make progress, we need to observationally characterize properties such as the strengths and topologies of galactic and intergalactic magnetic fields. I propose a portfolio of ground-breaking research projects that will capitalise on the recently upgraded Jansky VLA radio telescope to provide new observational measurements of interstellar and intergalactic magnetic fields and elucidate their roles in shaping the Universe. A Marie Skłodowska-Curie Actions Research Fellowship will facilitate movement from my current position as an Assistant Scientist at the National Radio Astronomy Observatory in the USA to ...</code>
  <code>identify mechanisms for building cross-cultural peace in frontier societies after European expansion</code>	<code>This research project investigates the historical roots of cross-cultural peacebuilding. European expansion during the sixteenth century was characterised by a high degree of violence, which was fuelled by cultural differences and religious radicalisms. The meeting of different cultures created new forms of violence, but, at the same time, generated new forms of cross-cultural encounters that were driven to construct a lasting peace. This project explains the transition from the violent conflicts of the first encounters between Europeans and non-European peoples, to their peaceful coexistence. Rather than focusing on cross-cultural diplomacy or treaty-making, this research project explains how peace was constructed “on the ground”. In order to do so, it focuses on three different frontiers of the first global empires. Through a comparative survey that includes cases from the Mediterranean, the Americas and Asia, this project shows that frontier societies resulting from European expansi...</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "CachedMultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          384,
          128
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• warmup_steps: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 128
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: no
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Training Time

• Training: 17.7 minutes

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.4.0
• Transformers: 5.5.3
• PyTorch: 2.7.1+cu118
• Accelerate: 1.13.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->