heart-e5

StrictSentenceTransformer based on intfloat/e5-base-v2

This is a sentence-transformers model finetuned from intfloat/e5-base-v2 on the bge-multilingual-gemma2-data dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-base-v2 <!-- at revision f52bf8ec8c7124536f0efb74aca902b2995e5bcd -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • bge-multilingual-gemma2-data

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

StrictSentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vaktibabat/heart-e5")
# Run inference
sentences = [
    'query: calories in chocolate covered pretzel',
    'passage: Chocolate Covered Pretzels There are 140 calories in a 2 pretzels serving of Sarris Candies Chocolate Covered Pretzels. Calorie breakdown: 39% fat, 55% carbs, 6% protein.',
    'passage: Calories In Doughnut - Glazed, Dunking, Cake, Chocolate The average amount of calories in a chocolate doughnut is approximately 340 calories. In addition to its high calorie content, chocolate doughnuts are also high in fats and processed sugars. Make sure that you check the nutritional breakdown of a chocolate doughnut box before you dig in!',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9357, 0.8240],
#         [0.9357, 1.0000, 0.9000],
#         [0.8240, 0.9000, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

My Information Retrieval

• Datasets: train_subset and dev_subset
• Evaluated with <code>training.train_utils.my_sentence_transformers.MyInformationRetrievalEvaluator.MyInformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

bge-multilingual-gemma2-data

• Dataset: bge-multilingual-gemma2-data at ef165e1
• Size: 1,534,495 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 6 tokens</li><li>mean: 10.85 tokens</li><li>max: 36 tokens</li></ul>	<ul><li>min: 21 tokens</li><li>mean: 85.96 tokens</li><li>max: 206 tokens</li></ul>	<ul><li>min: 21 tokens</li><li>mean: 86.98 tokens</li><li>max: 237 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>query: lee price wrestler</code>	<code>passage: - April 15, 2017. Lee Price Star Wrestler. Lee Price, the blonde 1990’s female wrestling Icon was fun to watch and seemed to be even more enjoyable to be around. As we all know, people who take life too seriously are no fun to be around especially if they are family or friends.</code>	<code>passage: - JACKSONVILLE, Fla. – William Donovan Lee, 42, was found guilty as charged Thursday afternoon of two counts of Attempted Murder in the First Degree, one count of Shooting or Throwing Deadly Missiles, and one count of Tampering with….</code>
  <code>query: what type of soil are crops grown</code>	<code>passage: Types of Soils Soil types according to depth are as follows: 1) Shallow Soil - Soil depth less than 22.5cm. Only shallow rooted crops are grown in such soil, e.g. Paddy, Nagli. 2) Medium deep soil - Soil depth is 22.5 to 45cm. Crops with medium deep roots are grown in this type of soil e.g. Sugar cane, Banana, Gram. 3) Deep soil - Soil depth is more than 45cm. Crops with long and deep roots are grown in this type a soil e.g.</code>	<code>passage: - Cover crops usually are grown to prevent soil loss from wind and water erosion. Use fast-growing cover crops, such as winter wheat or annual rye, on fall-spaded gardens. A second, and probably more important reason home gardeners should use cover crops is to improve soil structure and increase organic matter.</code>
  <code>query: how long does lisinopril stay in your body</code>	<code>passage: How long does it take for lisonopril to get out of your body after stopping dose? Responses (1) The plasma half-life of lisinopril is approximately 12 hours. This means that in the average person, lisinopril should disappear from the body within 3-4 days. However, since some of the effects of the drug last longer than that, the effects of this drug can last up to 1-2 weeks after stopping the drug. Take care.</code>	<code>passage: Top 30 Doctor insights on: How Long Does Lisinopril Stay In Your System Not sure if: your symptoms are side effects of your new medication. It usually takes no more than 1 or 2 days for any drug to clear from your body, but it is a gradual process. Other potential problematic scenariors: Drug metabolites can hang around for longer, tissue drug levels can sometimes take longer than blood levels, any damage that it might have caused can take longer to heal. Consult your doctor. ...Read more.</code>

• Loss: <code>training.train_utils.my_sentence_transformers.MyCachedMultipleNegativesRankingLoss.MyCachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 100.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 96,
      "gather_across_devices": false
  }
  

Evaluation Dataset

bge-multilingual-gemma2-data

• Dataset: bge-multilingual-gemma2-data at ef165e1
• Size: 7,712 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 6 tokens</li><li>mean: 10.99 tokens</li><li>max: 32 tokens</li></ul>	<ul><li>min: 25 tokens</li><li>mean: 87.85 tokens</li><li>max: 235 tokens</li></ul>	<ul><li>min: 22 tokens</li><li>mean: 87.77 tokens</li><li>max: 205 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>query: what is dornase alfa meaning</code>	<code>passage: - Dornase alfa is a biosynthetic form of human DNase I. The enzyme is involved in endonucleolytic cleavage of extracellular DNA to 5´-phosphodinucleotide and 5´-phosphooligonucleotide end products.It has no effect on intracellular DNA.tudies in rats indicate that, following aerosol administration, the disappearance half-life of dornase alfa from the lungs is 11 hours. In humans, sputum DNase levels declined below half of those detected immediately post-administration within 2 hours but effects on sputum rheology persisted beyond 12 hours.</code>	<code>passage: âDead on arrivalâ DOA or dead on arrival traditionally is the official terminology used by the trauma center where a victim is received. Trauma centers (emergency rooms) are where the official pronouncement (declaration) of death is made by the attending physician, and it is pronounced at the time the victim arrives at the facility.</code>
  <code>query: what is a domain name?</code>	<code>passage: Domain name A domain name is an identification string that defines a realm of administrative autonomy, authority or control within the Internet. Domain names are formed by the rules and procedures of the Domain Name System (DNS). Any name registered in the DNS is a domain name.Domain names can also be thought of as a location where certain information or activities can be found. Domain names are used in various networking contexts and application-specific naming and addressing purposes. fictitious domain name is a domain name used in a work of fiction or popular culture to refer to a domain that does not actually exist, often with invalid or unofficial top-level domains such as .web , a usage exactly analogous to the dummy 555 telephone number prefix used in film and other media.</code>	<code>passage: - The Internet Domain Name System (DNS) is the Internet’s hierarchical address. system that helps users find targeted webpages. The DNS functions like a telephone. directory for the Internet; a domain name is the user -friendly form of a webpage’s. “phone number,” which is called the Internet Protocol (IP) address.</code>
  <code>query: harmful effects of inflammation</code>	<code>passage: Doctor speaks on health effects of chronic inflammation Doctor speaks on health effects of chronic inflammation. If your body is in a chronic state of inflammation, it can have serious effects on your cellular health, and has been linked to degenerative diseases including cancer, heart disease, diabetes, Alzheimer's and many others.</code>	<code>passage: 10 Top Foods That Prevent Inflammation in Your Body And if you want to get or remain healthy, you definitely want to reduce the damaging effects of it! Inflammation has a positive and negative affect in your body. Inflammation has a positive side because it helps your body respond to stress. But chronic low-grade inflammation is thought to be one of the leading causes of disease, premature aging and illness. When you get a cold, your body responds with inflammation in the form of a fever that helps you heal.</code>

• Loss: <code>training.train_utils.my_sentence_transformers.MyCachedMultipleNegativesRankingLoss.MyCachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 100.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 96,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• learning_rate: 2e-05
• weight_decay: 0.01
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• dataloader_num_workers: 15
• load_best_model_at_end: True
• gradient_checkpointing: True
• eval_on_start: True

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 15
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: True
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.1.0
• Transformers: 4.57.1
• PyTorch: 2.9.1+cu128
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MyCachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->