ModernBERT-neg-sampling-PubMed

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the soda-vec-data-full_pmc_title_abstract_paired dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base <!-- at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8 -->
• Maximum Sequence Length: 768 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • soda-vec-data-full_pmc_title_abstract_paired
• Language: en

<!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 768, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("EMBO/negative_sampling")
# Run inference
sentences = [
    'Effects of vagus nerve stimulation on rat hippocampal progenitor proliferation.',
    'Vagus nerve stimulation (VNS), used in the treatment of epilepsy, was approved recently for treatment-resistant depression. The mechanisms of action of the VNS anti-depressive effects are not yet fully elucidated. Modulation of hippocampal neurogenesis has been proposed as an important factor in depression pathogenesis. We evaluated the effects of VNS on hippocampal progenitor turnover in the adult rat brain. Rats receiving VNS at the output current of 0.75 mA VNS for 2 days showed a significant 50% increase in dentate gyrus BrdU-incorporation consistent with an increase in progenitor proliferation. Output currents of 0.5 or 1.5 mA yielded non-significant trends for increased BrdU-labeling indicating an inverted U-shaped proliferative dose response to VNS as previously reported for other VNS-induced effects. Specific analysis for progenitor survival revealed no effects by VNS on dentate gyrus BrdU-labeling. These results suggest that VNS induced an increase in the number of available progenitor cells in the adult rat dentate gyrus by a mechanism presumably involving increased progenitor proliferation.',
    '18804463',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.8059, -0.1080],
#         [ 0.8059,  1.0000,  0.1485],
#         [-0.1080,  0.1485,  1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

soda-vec-data-full_pmc_title_abstract_paired

• Dataset: soda-vec-data-full_pmc_title_abstract_paired at 309bc96
• Size: 26,209,161 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>pmcid</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	pmcid
  type	string	string	string
  details	<ul><li>min: 6 tokens</li><li>mean: 24.12 tokens</li><li>max: 112 tokens</li></ul>	<ul><li>min: 27 tokens</li><li>mean: 285.54 tokens</li><li>max: 768 tokens</li></ul>	<ul><li>min: 4 tokens</li><li>mean: 5.31 tokens</li><li>max: 6 tokens</li></ul>

• Samples:

  anchor	positive	pmcid
  <code>Development and validation of a fast static headspace GC method for determination of residual solvents in permethrin.</code>	<code>A fast static headspace gas chromatography (HS-GC) method was developed to separate all residual solvents present in commercial active pharmaceutical ingredient (API) batches of permethrin. A total of six residual solvents namely 2-methylpentane, 3-methylpentane, methylcyclopentane, n-hexane, cyclohexane and toluene were found in typical commercial batches of permethrin; and three of them are not in the list of ICH solvents. All six residual solvents were baseline separated in five minutes by the new method presented in this paper. The method was successfully validated as per International Conference on Harmonisation (ICH) guidelines. Evaluation of this method was conducted to separate 26 commonly used solvents in the manufacturing of various APIs, key intermediates of APIs and pharmaceutical excipients. The results of the evaluation demonstrated that this method can also be used as a general method to determine residual solvents in various APIs, intermediates and excipients that are u...</code>	<code>27344630</code>
  <code>Moderating laboratory adaptation with the use of a heart-rate variability biofeedback device (StressEraser).</code>	<code>Difficulty sleeping is a common problem with laboratory polysomnograms. This affects both polysomnograms that are used as a clinical tool to investigate sleep pathology or as an outcome variable in research. The goal of this study was to use a handheld biofeedback device (StressEraser) to improve sleep quality in the laboratory. Ten subjects without a history of sleep disorders were randomly assigned to either a StressEraser or no-treatment control condition. A sleep disturbance scale derived from sleep efficiency, REM latency, minutes of stage 1 sleep, and wake after sleep onset was created to evaluate the differences between these groups. Subjects in the StressEraser group had significantly lower scores on the sleep disturbance scale compared to the no-treatment control group (p = 0.003). Sleep latency was not improved. In conclusion, the StressEraser significantly improved sleep quality compared to a no-treatment control group. This suggests that the StressEraser may be an effective...</code>	<code>19418214</code>
  <code>Stress induced phase transformation of a cesium stabilized leucite porcelain and associated properties.</code>	<code>The addition of CS2O to dental porcelains might provide a means for controlling the thermal expansion and toughness of feldspathic porcelains. The purpose of this investigation was to determine for a leucite porcelain the effect of CS2O content upon its coefficient of thermal expansion (alpha), toughness, hardness, and content of low (tetragonal) leucite and high (cubic) leucite. In order to determine the amount of low leucite in the specimens, an x-ray calibration curve for low leucite and an internal standard of copper was obtained using quantitative x-ray diffraction techniques. Utilizing a stress induced phase transformation between low and high leucite, an x-ray intensity conversion ratio (r) was determined for high leucite so that the calibration curve for low leucite could be used to determine the amount of high leucite present in the experimental porcelains. Specimens were prepared with various amounts of CS2O (0.0, 0.5, 1.0, 1.5, 2.0 mol%) so that, except for the as-received p...</code>	<code>10196797</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

soda-vec-data-full_pmc_title_abstract_paired

• Dataset: soda-vec-data-full_pmc_title_abstract_paired at 309bc96
• Size: 1,000 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>pmcid</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	pmcid
  type	string	string	string
  details	<ul><li>min: 7 tokens</li><li>mean: 23.68 tokens</li><li>max: 59 tokens</li></ul>	<ul><li>min: 29 tokens</li><li>mean: 277.39 tokens</li><li>max: 768 tokens</li></ul>	<ul><li>min: 4 tokens</li><li>mean: 5.32 tokens</li><li>max: 6 tokens</li></ul>

• Samples:

  anchor	positive	pmcid
  <code>Early and intermediate follow-up results after treatment of degenerative disc disease with the Bryan cervical disc prosthesis: single- and multiple-level.</code>	<code>Our clinical study design was prospective, concurrently enrolled and single-center trial of the new functional intervertebral cervical disc prosthesis (Medtronic Sofamor Danek, Memphis, TN) in the treatment of patients with single-level and multiple-level degenerative disc disease of the cervical spine. The study was designed to investigate the surgical technology skills and clinical effects of Bryan cervical disc prosthesis in Chinese, and to observe the stability and range of movement in the early and immediate postoperative period. Recently, motion preservation has come to the forefront of emerging technologies in spine surgery. This is the important background information of the emergence of cervical arthroplasty as an alternative to arthrodesis that offers the promise of restoring normal spinal movement and reduces a kinematic strain on adjacent segments. Nineteen patients (23 discs) had spinal arthroplasty with placement of the Bryan cervical artificial disc prosthesis in our hos...</code>	<code>18496332</code>
  <code>Decreased sympatho-adrenal activity in diabetic patients with autonomic dysfunction following mental stress.</code>	<code>The sympathetic nervous system is of major importance in the regulation of numerous physiological functions. While it is clearly established that there is a decreased noradrenergic status in people with autonomic neuropathy, the epinephrine secretion is much more controversial. Basal and mental stress-stimulated plasma catecholamine levels were measured in 42 diabetic patients with and without pathological cardiovascular function tests and in 13 healthy, non-diabetic control subjects. In addition, the excretion of catecholamines was measured in a 24 h urine collection and compared with the stress stimulated plasma levels. During mental stress exposure a diminished epinephrine secretion was found in diabetic patients with autonomic neuropathy compared with diabetic patients without neuropathy and the healthy control group (p < 0.05: respectively). The decreased epinephrine response to mental stress was strongly correlated with a diminished urinary excretion of this "neurotransmitter' (r...</code>	<code>8912251</code>
  <code>Hamburger-associated Escherichia coli O157:H7 infection in Las Vegas: a hidden epidemic.</code>	<code>This study sought to determine whether a multistate fast food hamburger-associated outbreak of Escherichia coli O157:H7 infection involved Las Vegas residents as well and, if so, why public health officials had not detected it. A matched case-control study was conducted among persons with bloody diarrhea and their healthy meal companions. Hamburger production, distribution, and cooking methods were reviewed. Unused hamburger patties were cultured, and E. coli O157:H7 isolates were characterized. Local laboratory stool culture practices were reviewed. Fifty-eight cases of bloody diarrhea were identified. Illness was associated with eating regular hamburgers (matched odds ratio [OR] = 9.0, 95% confidence interval [CI] = 1.02,433.4), but 25% of ill persons reported eating only jumbo hamburgers. Regular and jumbo hamburger patties yielded E. coli O157:H7 indistinguishable from the lone clinical isolate. No local laboratory cultured routinely for E. coli O157:H7 until after the outbreak. A ...</code>	<code>9103093</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• weight_decay: 0.01
• num_train_epochs: 1
• warmup_ratio: 0.1
• warmup_steps: 100
• fp16: True
• dataloader_num_workers: 4
• push_to_hub: True
• hub_model_id: EMBO/negative_sampling

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 100
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: EMBO/negative_sampling
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• dataset_name: EMBO/soda-vec-data-full_pmc_title_abstract_paired
• model_name: answerdotai/ModernBERT-base
• early_stopping_patience: 3
• early_stopping_threshold: 0.0
• use_adaptive_evaluation: True
• loss_function: negative_sampling
• pooling_strategy: mean

</details>

Training Logs

<details><summary>Click to expand</summary>

The model card is truncated. Read the rest on Hugging Face.