nomic-embed-text-v1.5_Clinical-Trials_Matryoshka

nomic-embed text-v1.5 Clinical-Trials Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v1.5 on the dataset from Clinical_trials_anchor-positive-pairs_EmbeddingModel-data. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/nomic-embed-text-v1.5 <!-- at revision e9b6763023c676ca8431644204f50c2b100d9aab -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset: Clinical_trials_anchor-positive-pairs_EmbeddingModel-data
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NomicBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vab46/nomic-embed-text-v1.5_Clinical-Trials_Matryoshka")
# Run inference
documents = [
    "TITLE: Exergames-acceptance and Commitment Therapy(e-ACT) for Breast Cancer Depression and Anxiety: A Randomized Controlled Trial The Efficacy of Exergames-acceptance and Commitment Therapy Program for Treatment of Depression and Other Psychological Complications in Breast Cancer: Comparison With Acceptance and Commitment Therapy Alone and Treatment-as-usual in a Randomized Controlled Trial\nSUMMARY: The goal of this three arm, double-blind, randomized controlled trial is to learn if an exergame-acceptance and commitment therapy (e-ACT) program can treat depression and anxiety in breast cancer patients, compared to acceptance and commitment therapy (ACT) alone and treatment-as-usual. The main question it aims to answer is:\nIs there a difference in the effectiveness of the e-ACT program in reducing depressive and anxiety symptoms, cancer-related fatigue, experiential avoidance, and serum IL-6 levels, while increasing BDNF and improving posttraumatic growth and quality of life among breast cancer patients, compared to ACT alone and treatment-as-usual, measured at baseline, 8 weeks (post-intervention), and 12 weeks after the intervention (follow-up)?\nResearchers will compare the e-ACT group, the ACT-alone group, and the treatment-as-usual group to see if the e-ACT program yields superior outcomes in reducing psychological distress and improving well-being.\nParticipants will:\nBe randomly assigned to one of three groups: (1) e-ACT (exergame + ACT), (2) ACT alone, or (3) treatment-as-usual (general patient education).\nAttend an 8-week program (one session per week) if in the e-ACT or ACT group; the control group continues their usual care.\nComplete questionnaires at three time points (baseline, 8 weeks, and 20 weeks) to assess depression, anxiety, quality of life, posttraumatic growth, valued living, experiential avoidance, and cancer-related fatigue.\nProvide blood samples at pre-intervention and post-intervention (week 8) for analysis of interleukin-6 (IL-6) and brain-derived neurotrophic factor (BDNF) biomarkers.\nThe INCLUSION CRITERIA include:\n1. Newly diagnosed breast cancer patients and patients with recurrent breast cancer confirmed by histopathological report, regardless of the stage of cancer.\n2. Those with HADS score of 8 or higher in both Depression and Anxiety sub-scales of the HADS.\n3. Patients who had been treated with surgery or were undergoing the standard regime of clinical anti-tumor treatment (chemotherapy, radiotherapy, immunotherapy, targeted therapy, etc.).\n4. Age 18 years old and above.\n5. Patients who were able to read and understand written Chinese.\nThe EXCLUSION CRITERIA are:\n1. Pregnant women，considered for the following reasons: Pregnancy involves significant hormonal and physical changes that could affect the participant's response to the e-ACT; the safety of e-ACT to the unborn child was of concern.\n2. Those who have current and lifetime history of engaging in any psychotherapy\n3. Those who consumed alcohol and illicit drugs .\n4. Those who has current and lifetime history of other psychiatric illnesses, such as psychotic disorders (schizophrenia, schizophreniform disorder, schizoaffective disorders, brief psychotic disorder, and delusional disorder), bipolar mood disorder, obsessive compulsive disorder, posttraumatic stress disorder, and attention deficit hyperactive disorder, and autism spectrum disorder\n5. Those who are on medications that can induce psychiatric symptoms, such as cardiovascular agents (clonidine, guanethidine, methyldopa, reserpine, beta blockers), dermatologic agents (isotretinoin), anticonvulsants (levetiracetam), antimigraine medications (triptans), hormonal agents (corticosteroids, oral contraceptives, gonadotropin-releasing hormone agonists, tamoxifen), varenicline, immunological agents (interferons), and levodopa.Or those who ccurrently using any psychotropic medication.\n6. Patient who has suicidal tendency.\n7. those who are physically unfit to answer questionnaires. (those are bed-bound or too weak to answer the questionnaire).",
]
queries = [
    'Could a patient with a history of depression and anxiety, who has been diagnosed with breast cancer, qualify for this study?',
    'I have obesity and high blood pressure, can I participate in this study and what will happen if I join?',
    'How old do you need to be to be eligible for this clinical trial?',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7282, 0.1019, 0.2957]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768, dim_512, dim_256, dim_128, dim_64
• Evaluated with <code>InformationRetrievalEvaluator</code> with these parameters:

Comparison with the baseline model performance on same data

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* - Currently there are some **precision issues** particularly due to data noise(document chunks massively large at times). - Also there were 4 synthetic anchors(queries) generated from a positive(document). - So work in progress to shorten independent portions of document(i.e. from clinical trial records) and creating a mix mapping type between (query -> document), not just many to one relations. - Once chunks get more leaner and we have good proportion of one to one alongside many to one(anchor-positive) pair maps, precision shall greatly improve. --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* - For the time being the emv=bedding model is suited to tasks like knowing similarity bewteen a **short summary/ Long answer questions mapping with relevant document/output related to dug trials**. - These embeddings should are highly specific to pubmed , clinical trials related drug R&D tasks; so performance in other domains is subject to evaluation currently. -->

Training Details

Training Dataset

json

• Dataset: json
• Size: 7,193 training samples
• Columns: <code>positive</code> and <code>anchor</code>
• Approximate statistics based on the first 100 samples:

  	positive	anchor
  type	string	string
  modality	text	text
  details	<ul><li>min: 271 tokens</li><li>mean: 671.29 tokens</li><li>max: 2110 tokens</li></ul>	<ul><li>min: 10 tokens</li><li>mean: 20.11 tokens</li><li>max: 35 tokens</li></ul>

• Samples:

  positive	anchor
  <code>TITLE: Comparison of Postoperative Analgesic Efficacy of Two Different Blocks in Modified Radical Mastectomy Surgery Comparison of Postoperative Analgesic Efficacy of Superior Posterior Serratus Intercostal Plane Block (SPSIP) and Anterior Serratus Plane Block in Modified Radical Mastectomy Surgery<br>SUMMARY: Postoperative pain following modified radical mastectomy remains a significant clinical challenge and may delay recovery, impair patient comfort, and increase opioid consumption. Ultrasound-guided fascial plane blocks have become an important component of multimodal analgesia for breast surgery. The serratus anterior plane block (SAPB) is an established regional anesthesia technique that provides effective postoperative analgesia. The serratus posterior superior intercostal plane block (SPSIPB) is a recently described interfascial block with promising analgesic effects, but evidence comparing its efficacy with SAPB in breast surgery remains limited.<br>This prospective, randomized, dou...</code>	<code>Is this trial open to male patients?</code>
  <code>TITLE: Masticatory Efficiency, Bite Force, and Patient-Reported Outcomes in Patients Rehabilitated With Complete and Partial Dentures Supported by Teeth, Mini-Implants, or Conventional Implants Masticatory Efficiency, Bite Force, and Patient-Reported Outcomes in Patients Rehabilitated With Conventional Complete Dentures, Conventional Removable Partial Dentures, Mini-Implant-Retained Overdentures, Conventional Implant-Retained Overdentures, and Implant-Assisted Removable Partial Dentures: A Cross-Sectional and Longitudinal Clinical Study<br>SUMMARY: This study aims to evaluate masticatory efficiency, bite force, and patient-reported outcomes in edentulous and partially edentulous patients rehabilitated with different removable prosthodontic treatment modalities. The study includes both cross-sectional and longitudinal components. The cross-sectional component will compare patients wearing conventional complete dentures, conventional removable partial dentures, mini-implant-retained overden...</code>	<code>Could I qualify for this trial if I have a partial denture?</code>
  <code>TITLE: ctDNA-driven Adaptive Proton Craniospinal Irradiation in Non-Small Cell Lung Cancer With Leptomeningeal Metastasis After Resistance to Third-Generation TKIs in the Consolidation Phase Dynamic Adaptive Radiotherapy for Non-Small Cell Lung Cancer With Leptomeningeal Metastasis After Resistance to Third-Generation TKIs in the Consolidation Phase: A Multicenter Randomized Controlled Trial Comparing Outcomes Between Proton Craniospinal Irradiation and Intrathecal Pemetrexed (DART-LM)<br>SUMMARY: **Brief Summary (English)**<br>The goal of this clinical trial is to learn if a risk-adaptive consolidation therapy, guided by cerebrospinal fluid (CSF) circulating tumor DNA (ctDNA) clearance kinetics after induction intrathecal pemetrexed, can improve intracranial progression-free survival (iPFS) compared to standard intrathecal pemetrexed consolidation in patients with leptomeningeal metastasis (LM) from EGFR-mutant non-small cell lung cancer (NSCLC) that has progressed on third-generation E...</code>	<code>I have lung cancer that has spread to my brain, can I take part in a trial that uses a special radiation treatment?</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• num_train_epochs: 4
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• gradient_accumulation_steps: 8
• fp16: True
• per_device_eval_batch_size: 32
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 4
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 8
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

</details>

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.6 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.7.0
• Transformers: 5.15.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->