aie5-midter-new

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l <!-- at revision d8fb21ca8d905d2832ee8b96c894d3298964346b -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("christinemahler/aie5-midter-new")
# Run inference
sentences = [
    'What is the purpose of the funding opportunity RFA-DK-26-007 titled "Collaborative Research Using Biosamples and/or Data from Type 1 Diabetes Clinical Studies"?',
    'RFA-DK-26-007: Collaborative Research Using Biosamples and/or Data from Type 1 Diabetes Clinical Studies (R01 - Clinical Trial Not Allowed) Part 1. Overview Information\n    \n \n \n   Participating Organization(s)\n    \n    National Institutes of Health (\n     \n     NIH\n     \n    )\n    \n \n \n   Components of Participating Organizations\n    \n    National Institute of Diabetes and Digestive and Kidney Diseases (\n     \n     NIDDK\n     \n    )\n     \n \n     Office of The Director, National Institutes of Health (\n      \n      OD\n      \n     )\n     \n \n \n   Funding Opportunity Title\n    \n    Collaborative Research Using Biosamples and/or Data from Type 1 Diabetes Clinical Studies (R01 - Clinical Trial Not Allowed)\n    \n \n \n   Activity Code\n    \n \n     R01\n     \n    Research Project Grant\n    \n \n \n   Announcement Type\n   Reissue of\n    \n    RFA-DK-22-021\n    \n \n \n \n   Related Notices\n    \n \n \n      April 4, 2024\n      \n     - Overview of Grant Application and Review Changes for Due Dates on or after January 25, 2025. See Notice\n      \n      NOT-OD-24-084\n      \n     .\n     \n \n \n      August 31, 2022\n      \n     - Implementation Changes for Genomic Data Sharing Plans Included with Applications Due on or after January 25, 2023. See Notice\n      \n      NOT-OD-22-198\n      \n     .\n     \n \n \n      August 5, 2022\n      \n     - Implementation Details for the NIH Data Management and Sharing Policy. See Notice\n      \n      NOT-OD-22-189\n      \n     .\n     \n \n \n \n   Funding Opportunity Number (FON)\n    \n    RFA-DK-26-007\n    \n \n \n   Companion Funding Opportunity\n   None\n    \n \n   Number of Applications\n    \n    See\n     \n     Section III. 3. Additional Information on Eligibility\n     \n    .\n    \n \n \n   Assistance Listing Number(s)\n   93.847\n    \n \n   Funding Opportunity Purpose\n    \n    This Notice of Funding Opportunity (NOFO) invites applications for studies of type 1 diabetes etiology and pathogenesis using data and samples from clinical trials and studies. This opportunity is intended to fund investigative teams collaborating to answer important questions about disease mechanisms leading to improved delay and durable prevention of type 1 diabetes. This NOFO is associated with the Special Diabetes Program (\n     \n     https://www.niddk.nih.gov/about-niddk/research-areas/diabetes/type-1-diabetes-special-statutory-funding-program/about-special-diabetes-program\n     \n    ) which funds research on the prevention, treatment, and cure of type 1 diabetes and its complications, including unique, innovative, and collaborative research consortia and clinical trials networks.\n    \n \n \n \n \n \n \n   Funding Opportunity Goal(s)\n    \n    To promote extramural basic and clinical biomedical research that improves the understanding of the mechanisms underlying disease and leads to improved preventions, diagnosis, and treatment of diabetes, digestive, and kidney diseases. Programmatic areas within the National Institute of Diabetes and Digestive and Kidney Diseases include diabetes, digestive, endocrine, hematologic, liver, metabolic, nephrologic, nutrition, obesity, and urologic diseases.\n    \n This variable defines that we need to start a new row.',
    'RFA-DK-26-007: Collaborative Research Using Biosamples and/or Data from Type 1 Diabetes Clinical Studies (R01 - Clinical Trial Not Allowed) Part 1. Overview Information\n    \n \n \n   Participating Organization(s)\n    \n    National Institutes of Health (\n     \n     NIH\n     \n    )\n    \n \n \n   Components of Participating Organizations\n    \n    National Institute of Diabetes and Digestive and Kidney Diseases (\n     \n     NIDDK\n     \n    )\n     \n \n     Office of The Director, National Institutes of Health (\n      \n      OD\n      \n     )\n     \n \n \n   Funding Opportunity Title\n    \n    Collaborative Research Using Biosamples and/or Data from Type 1 Diabetes Clinical Studies (R01 - Clinical Trial Not Allowed)\n    \n \n \n   Activity Code\n    \n \n     R01\n     \n    Research Project Grant\n    \n \n \n   Announcement Type\n   Reissue of\n    \n    RFA-DK-22-021\n    \n \n \n \n   Related Notices\n    \n \n \n      April 4, 2024\n      \n     - Overview of Grant Application and Review Changes for Due Dates on or after January 25, 2025. See Notice\n      \n      NOT-OD-24-084\n      \n     .\n     \n \n \n      August 31, 2022\n      \n     - Implementation Changes for Genomic Data Sharing Plans Included with Applications Due on or after January 25, 2023. See Notice\n      \n      NOT-OD-22-198\n      \n     .\n     \n \n \n      August 5, 2022\n      \n     - Implementation Details for the NIH Data Management and Sharing Policy. See Notice\n      \n      NOT-OD-22-189\n      \n     .\n     \n \n \n \n   Funding Opportunity Number (FON)\n    \n    RFA-DK-26-007\n    \n \n \n   Companion Funding Opportunity\n   None\n    \n \n   Number of Applications\n    \n    See\n     \n     Section III. 3. Additional Information on Eligibility\n     \n    .\n    \n \n \n   Assistance Listing Number(s)\n   93.847\n    \n \n   Funding Opportunity Purpose\n    \n    This Notice of Funding Opportunity (NOFO) invites applications for studies of type 1 diabetes etiology and pathogenesis using data and samples from clinical trials and studies. This opportunity is intended to fund investigative teams collaborating to answer important questions about disease mechanisms leading to improved delay and durable prevention of type 1 diabetes. This NOFO is associated with the Special Diabetes Program (\n     \n     https://www.niddk.nih.gov/about-niddk/research-areas/diabetes/type-1-diabetes-special-statutory-funding-program/about-special-diabetes-program\n     \n    ) which funds research on the prevention, treatment, and cure of type 1 diabetes and its complications, including unique, innovative, and collaborative research consortia and clinical trials networks.\n    \n \n \n \n \n \n \n   Funding Opportunity Goal(s)\n    \n    To promote extramural basic and clinical biomedical research that improves the understanding of the mechanisms underlying disease and leads to improved preventions, diagnosis, and treatment of diabetes, digestive, and kidney diseases. Programmatic areas within the National Institute of Diabetes and Digestive and Kidney Diseases include diabetes, digestive, endocrine, hematologic, liver, metabolic, nephrologic, nutrition, obesity, and urologic diseases.\n    \n This variable defines that we need to start a new row.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 216 training samples
• Columns: <code>sentence_0</code> and <code>sentence_1</code>
• Approximate statistics based on the first 216 samples:

  	sentence_0	sentence_1
  type	string	string
  details	<ul><li>min: 19 tokens</li><li>mean: 34.9 tokens</li><li>max: 59 tokens</li></ul>	<ul><li>min: 8 tokens</li><li>mean: 377.44 tokens</li><li>max: 512 tokens</li></ul>

• Samples:

  sentence_0	sentence_1
  <code>What initiatives is the Department of Health and Human Services pursuing under opportunity ID [insert ID] to improve public health outcomes?</code>	<code>Department of Health and Human Services</code>
  <code>How does the title of opportunity ID [insert ID] align with the strategic goals of the Department of Health and Human Services?</code>	<code>Department of Health and Human Services</code>
  <code>What are the main goals of the funding opportunity titled "Laboratory Flexible Funding Model (LFFM)" under opportunity ID RFA-FD-25-007?</code>	<code>RFA-FD-25-007: Laboratory Flexible Funding Model (LFFM) Part 1. Overview Information<br> <br> <br> <br> Participating Organization(s)<br> <br> U.S. Food and Drug Administration (<br> <br> FDA<br> <br> )<br> <br> <br> <br> <br> NOTE: The policies, guidelines, terms, and conditions stated in this Notice of Funding Opportunity (NOFO) may differ from those used by the NIH.  Where this NOFO provides specific written guidance that may differ from the general guidance provided in the grant application form, please follow the instructions given in this NOFO.<br> <br> <br> The FDA does not follow the NIH Page Limitation Guidelines or the NIH Review Criteria. Applicants are encouraged to consult with FDA Agency Contacts for additional information regarding page limits and the FDA Objective Review Process.<br> <br> <br> <br> Components of Participating Organizations<br> <br> FOOD AND DRUG ADMINISTRATION (<br> <br> FDA<br> <br> )<br> <br> <br> <br> Funding Opportunity Title<br> <br> Laboratory Flexible Funding Model (LFFM)<br>...</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

</details>

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->