4_modernbert-embed-base

SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the ssf-train-valid-combi-v1v2v3 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base <!-- at revision d556a88e332558790b210f7bdbe87da2fa94a8d8 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ssf-train-valid-combi-v1v2v3

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("frankwong2001/4_modernbert-embed-base")
# Run inference
sentences = [
    'The Assistant Pastry Cook/Assistant Baker/Kitchen Assistant is responsible for the production of pastry and baked goods. He/She prepares the baking equipment and ingredients, and applies finishing touches in post-production of pastries and baked goods. He follows hygiene, safety and other standards, and carries out food and beverage operational tasks. He may suggest areas for continuous improvement within his own workstation. Attentive and meticulous, he possesses good time management skills and is able to multi-task, while performing physical tasks in a high-volume production environment. He is able to work under high temperatures, and in a flexible schedule, including weekends, evenings, and public holidays.',
    'The Assistant Pastry Cook/Assistant Baker/Kitchen Assistant is tasked with producing a variety of pastries and baked goods. He/She prepares the necessary baking equipment and ingredients, and adds finishing touches during the post-production phase of the items. He adheres to hygiene, safety, and other relevant standards, while executing food and beverage operational duties. He may also identify opportunities for continuous improvement in his workstation. Detail-oriented and diligent, he demonstrates strong time management skills and the ability to multi-task in a fast-paced production setting. He is capable of working in high temperatures and maintains a flexible schedule, which may include weekends, evenings, and public holidays.',
    'The Assistant Chef/Line Cook/Kitchen Staff is responsible for the preparation of savory dishes and entrees. He/She sets up the cooking appliances and ingredients, and applies garnishes during the serving of dishes. He follows cleanliness, safety, and other protocols, and performs kitchen and dining operational duties. He may recommend changes for better efficiency within his own cooking area. Attentive and thorough, he exhibits strong organizational skills and is adept at prioritizing tasks, while executing routine duties in a low-volume service environment. He is able to work in cold conditions, and adheres to a rigid schedule, which excludes weekends, evenings, and public holidays.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9323, 0.2374],
#         [0.9323, 1.0000, 0.2841],
#         [0.2374, 0.2841, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

ssf-train-valid-combi-v1v2v3

• Dataset: ssf-train-valid-combi-v1v2v3 at cf63f9b
• Size: 10,556 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 63 tokens</li><li>mean: 168.29 tokens</li><li>max: 355 tokens</li></ul>	<ul><li>min: 66 tokens</li><li>mean: 161.05 tokens</li><li>max: 317 tokens</li></ul>	<ul><li>min: 35 tokens</li><li>mean: 175.65 tokens</li><li>max: 1455 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>Lighting Designers are responsible for crafting lighting designs in line with the creative vision of the production. They are responsible for designing all aspects of lighting; from creating plans and specifying equipment placement to cue development, including brightness, colour and transitions. They are also responsible for all paperwork related to lighting, such as hook-ups, schedules and colour lists. They create lighting plots specifying the placement and configuration of all instruments and oversee lighting during production runs. Lighting Designers lead the lighting team and coordinate the development, installation and operation of the lighting design and any other special electrical effects. They decide on the lighting equipment to use from existing inventory or on the lighting rental package for venues with cold rig. They work in tandem with the creative leadership, other production designers and the lighting team to ensure the lighting complements all creative elements of the...</code>	<code>The Lighting Designer is tasked with developing innovative lighting concepts that align with the artistic direction of the performance. This role involves designing every detail of the lighting setup, from drafting plans and determining equipment locations to developing cues that encompass brightness, color, and transitions. Additionally, the Lighting Designer manages all documentation related to lighting, including connection details, schedules, and color charts. They create comprehensive lighting plots that outline the arrangement and configuration of all lighting fixtures and supervise the lighting during live performances. Leading the lighting team, the Lighting Designer coordinates the design, setup, and operation of the lighting scheme, as well as any unique electrical effects. They make informed decisions regarding the lighting gear to be utilized from the available inventory or select rental packages for venues with limited equipment. Collaborating closely with the creative tea...</code>	<code>The Lighting Technician is responsible for maintaining the sound systems in line with the overall production requirements. They handle all aspects of audio setup, from developing sound plans and specifying equipment placement to cue development, including volume, effects, and transitions. They also manage all paperwork related to sound, such as connection diagrams, schedules, and effect lists. The Lighting Technician creates sound plots specifying the placement and configuration of all audio instruments and oversees sound during rehearsal runs. They lead the sound team and coordinate the development, installation, and operation of the audio design and any other special sound effects. They decide on the audio equipment to use from existing inventory or on the sound rental package for venues with limited resources. They work closely with the technical team to ensure the sound complements all technical elements of the production.</code>
  <code>The Associate Director (Facilities Management) is responsible for driving strategies to improve facility operations. He/She builds strategic relationships with stakeholders and drives service excellence. He formulates organisational Workplace Safety and Health (WSH) practices as well as the green building strategies to fulfil environmental sustainability regulations. He is in charge of approving tender specifications, awarding works to selected bidders and endorsing contracts. He oversees the teams' development and recruitment and is responsible for the departments' financial planning and risk management. He is a subject matter expert and possesses excellent negotiation and people management skills. He is able to influence and communicate effectively with internal and external stakeholders.</code>	<code>The Associate Director (Facilities Management) plays a pivotal role in enhancing the efficiency of facility operations. This position involves cultivating strong partnerships with key stakeholders and championing exceptional service delivery. The individual is tasked with developing organizational practices for Workplace Safety and Health (WSH) alongside implementing sustainable green building initiatives to meet environmental regulations. Responsibilities also include approving tender proposals, selecting contractors for projects, and endorsing contractual agreements. Additionally, the Associate Director oversees team growth and recruitment efforts, manages the department's financial strategies, and addresses risk management. A recognized expert in the field, the individual demonstrates outstanding negotiation and interpersonal skills, effectively influencing and engaging with both internal and external parties.</code>	<code>The Associate Director (Event Management) coordinates various events and activities, focusing on logistics and attendee engagement. <br><br>The Associate Director (Facilities Management) is responsible for overseeing entry-level staff and managing basic office supplies without strategic oversight.<br><br>The Associate Director (Quality Assurance) ensures compliance with industry standards in a manufacturing environment, focusing on product testing and regulatory reporting.<br><br>The Associate Director (Facilities Management) manages operations in a multicultural context, aligning with international regulations and cross-border practices.<br><br>The Associate Director (Project Management) combines project planning and client relations, tasked with both strategic oversight and detailed operational execution across multiple projects.</code>
  <code>The Derivatives Trading Manager/Senior Derivatives Trader assumes responsibilities of quantifiable derivative trading portfolios and their supporting activities. He/She plans derivative trading activities that support his portfolio objectives and take leads in structured products' deal making. He is expected to develop derivative portfolio strategies to guide positions during various market trends or economic conditions. Armed with strong numerical and business acumen, he possesses a good understanding of market conditions as well.</code>	<code>The Derivatives Trading Manager is responsible for managing quantifiable derivative trading portfolios and their associated activities. He/She plans trading initiatives that align with portfolio objectives and takes the lead in structuring product deals. He is expected to formulate derivative portfolio strategies to navigate positions during diverse market trends or economic circumstances. Equipped with strong numerical skills and business insight, he has a solid grasp of market dynamics as well.</code>	<code>The Derivatives Trading Assistant manages non-quantifiable trading activities and their unrelated tasks. He/She organizes trading schedules that do not align with portfolio goals and takes a supportive role in unstructured product negotiations. He is expected to assist in developing non-derivative strategies to navigate decisions during unrelated market fluctuations or economic situations. Lacking strong numerical skills, he has minimal understanding of market dynamics as well.</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

ssf-train-valid-combi-v1v2v3

• Dataset: ssf-train-valid-combi-v1v2v3 at cf63f9b
• Size: 2,639 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 57 tokens</li><li>mean: 170.84 tokens</li><li>max: 352 tokens</li></ul>	<ul><li>min: 55 tokens</li><li>mean: 163.57 tokens</li><li>max: 319 tokens</li></ul>	<ul><li>min: 20 tokens</li><li>mean: 189.74 tokens</li><li>max: 1282 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>The General Manager/Managing Director/Vice President (Aircraft Maintenance) is responsible for defining the long-term strategic direction to grow the business in line with the organisations overall vision, mission and values. He/She promotes strategic aircraft maintenance programmes for business competitiveness and sets direction for leading aerospace maintenance practices in the organisation. He represents the organisation with customers, investors, and business partners, and holds responsibility for promoting organisational compliance with airworthiness and legislative requirements, fostering a culture of workplace safety and health, and championing leading practices and quality and risk management. He inspires the organisation towards achieving business goals by striving for continuous improvement, driving digital innovation and evaluating the organisation's approach towards a lean and sustainable enterprise. He demonstrates excellent leadership capabilities and builds strategic par...</code>	<code>The General Manager/Managing Director/Vice President (Aircraft Maintenance) is tasked with shaping the long-term strategic vision to enhance business growth in alignment with the organization's core values and mission. He/She advocates for strategic aircraft maintenance initiatives to ensure business competitiveness and provides direction for leading aerospace maintenance practices within the organization. He represents the organization to customers, investors, and business partners, while also ensuring compliance with airworthiness and legislative standards, promoting a culture of workplace safety and health, and championing best practices in quality and risk management. He motivates the organization to reach its business objectives through continuous improvement, driving digital innovation, and assessing the organization's commitment to a lean and sustainable enterprise. He showcases strong leadership skills and cultivates strategic partnerships with stakeholders to advance business ...</code>	<code>The Software Engineer develops innovative applications and ensures software quality through rigorous testing and debugging processes. He/She collaborates with design teams to create user-friendly interfaces and integrates feedback to enhance functionality and performance. He is responsible for maintaining code repositories and documenting software processes while staying updated with the latest technological advancements. He also participates in team meetings to discuss project progress and brainstorm new ideas for product improvement.</code>
  <code>The Associate User Interface Designer performs requirements analysis for the design of user interfaces (UIs) and drafts technical specifications for the design of UIs. He/She assists in the development and programming of intuitive and responsive UIs for each screen or page with which a user interacts. He assists in developing prototypes for UIs, conducts usability testing for validation, and supports the evaluation of the effectiveness of the UI. He prepares reports on UI design performance indicators, proposes, modifications in the design of user interface based on user feedback, as well as solutions to address design issues. He works in a team and is familiar with programming languages used by the organisation to design and develop UIs. He is familiar with graphic designing tools, and is also knowledgeable of Universal Principles of Design as well as commonly used design methods. The Associate UI Designer adopts a broad perspective to user interface design concepts, and is open to ex...</code>	<code>The Associate User Interface Designer is responsible for conducting requirements analysis to create user interfaces (UIs) and drafting technical specifications for UI design. He/She contributes to the development and programming of intuitive and responsive UIs for each user interaction screen or page. He assists in creating prototypes for UIs, performs usability testing for validation, and supports the assessment of UI effectiveness. He prepares reports on UI design performance metrics, suggests modifications based on user feedback, and offers solutions to design challenges. He collaborates within a team and is proficient in the programming languages utilized by the organization for UI design and development. He is knowledgeable in graphic design tools and well-versed in Universal Principles of Design and common design methodologies. The Associate UI Designer takes a comprehensive view of user interface design concepts and is eager to explore innovative options in the development of so...</code>	<code>The Associate Software Engineer conducts requirements analysis for the development of software applications and drafts technical specifications for application functionality. He/She assists in the coding and debugging of efficient and scalable applications for various platforms. He aids in creating application prototypes, performs functionality testing for validation, and supports the analysis of application performance metrics. He prepares reports on application development indicators, suggests enhancements based on user input, and provides solutions to functionality issues. He works within a team and is familiar with programming languages employed by the organization for software development. He is experienced with database management tools and knowledgeable of Software Development Life Cycle methodologies as well as commonly used programming practices. The Associate Software Engineer adopts a narrow focus on application development concepts and is hesitant to explore new approaches ...</code>
  <code>The Senior Project Engineer is responsible for executing project management plans from start to finish, to ensure project completions on time, and within budget. He/She typically comes from an engineering background with work experience in production and/or design, and is able to develop project schedules, budgets and manage project staff and subcontractors. He has good communication and negotiation skills for engaging internal and external parties to secure specialised resources and contributions for projects, and managing ongoing relationships with sub-contractors. He oversees sub-contractors schedules, performance, and payments, and has the responsibility to reschedule and coordinate work to ensure compliance with applicable project schedules.</code>	<code>The Senior Project Engineer plays a pivotal role in implementing comprehensive project management strategies from inception to completion, ensuring that projects are delivered on time and within financial constraints. This position typically requires a background in engineering, along with relevant experience in production and design. The Senior Project Engineer is adept at creating project timelines and budgets while effectively managing project teams and subcontractors. Strong communication and negotiation skills are essential for collaborating with both internal and external stakeholders to secure specialized resources and maintain productive relationships with subcontractors. Additionally, this role involves monitoring subcontractor schedules, performance metrics, and payment processes, along with the responsibility to adjust and coordinate work to align with project timelines.</code>	<code>The Senior Project Engineer is tasked with overseeing the financial audits of various departments, ensuring compliance with internal policies and external regulations. <br><br>The Senior Project Engineer is responsible for mentoring junior engineers while managing large-scale engineering projects, requiring at least ten years of experience in a senior leadership role.<br><br>The Senior Project Engineer focuses on compliance checks within the healthcare sector, utilizing analytical skills to assess regulatory frameworks and implement necessary changes.<br><br>The Senior Project Engineer is involved in developing marketing strategies for consumer products, requiring expertise in market analysis and brand management across international markets.<br><br>The Senior Project Engineer combines responsibilities of a project manager and a quality assurance officer, overseeing project execution while also conducting product testing and compliance evaluations.</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->