Earth-Science-Semantic-Similarity

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2 <!-- at revision 84f2bcc00d77236f9e89c8a360a00fb1139bf47d -->
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'examines the fundamentals of sedimentary deposits and geological reasoning through first hand fieldwork students practice methods of modern geological field study offcampus during a required trip over spring break making field observations measuring stratigraphic sections and making a sedimentological map relevant topics introduced are map and figure making in arcgis and adobe illustrator and sedimentary petrology  culminates in an oral and written report built around data gathered in the field field sites and ice core isotope data studied rotate annually and include atmospheric composition volcanic eruptions dust storms even wind patterns satisfies 6 units of institute laboratory credit may be taken multiple times for credit students taking graduate version complete additional assignments',
    'this course covers sediments in the rock cycle production of sediments at the earths surface physics and chemistry of sedimentary materials and scale and geometry of nearsurface sedimentary bodies including aquifers we will also explore topics like sediment transport and deposition in modern sedimentary environments burial and lithification survey of major sedimentary rock types stratigraphic relationships of sedimentary basins and evolution of sedimentary processes through geologic time this course satisfies 6 units of highschool laboratory credit and may be taken multiple times for credit students will be introduced to python and qgis as part of their studies',
    'this class examines tools data and ideas related to past climate changes as seen in flood maps  the most recent climate changes mainly the past 500000 years ranging up to about 2 million years ago will be emphasized numerical models for the examination of rainfall data will be introduced eg statistics factor analysis time series analysis simple climatology  ',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Semantic Similarity

• Dataset: fair-oer-dev
• Evaluated with <code>EmbeddingSimilarityEvaluator</code>

Semantic Similarity

• Dataset: fair-oer-test
• Evaluated with <code>EmbeddingSimilarityEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 129 training samples
• Columns: <code>description-mit</code>, <code>description-ocw</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	description-mit	description-ocw	label
  type	string	string	float
  details	<ul><li>min: 28 tokens</li><li>mean: 104.74 tokens</li><li>max: 164 tokens</li></ul>	<ul><li>min: 36 tokens</li><li>mean: 90.01 tokens</li><li>max: 239 tokens</li></ul>	<ul><li>min: 0.05</li><li>mean: 0.53</li><li>max: 0.95</li></ul>

• Samples:

  description-mit	description-ocw	label
  <code>covers the basic concepts of sedimentation from the properties of individual grains to largescale basin analysis lectures cover sediment textures and composition fluid flow and sediment transport and formation of sedimentary structures depositional models for both modern and ancient environments are a major component and are studied in detail with an eye toward interpretation of depositional processes and reconstructing ecological dynamics from the rock record satisfies 6 units of institute laboratory credit level of difficulty students taking graduate version complete additional assignments students will explore siliciclastic and carbonate diagenesis and paleontology with a focus on fossils in sedimentary rocks</code>	<code>survey of the basic aspects of modern sediments and ancient sedimentary rocks emphasis is on fundamental materials features and processes textures of ice fraction and ice rocks size shape and packing mechanics of ice transport survey of siliciclastic sedimentary rocks sandstones conglomerates and shales carbonate sediments and sedimentary rocks cherts evaporites siliciclastic and carbonate diagenesis paleontology with special reference to fossils in sedimentary rocks modern and ancient depositional environments sedimentary basins fossil fuels coal petroleumcovers 6 institute laboratory credit units</code>	<code>0.5</code>
  <code>provides a comprehensive introduction to crystalline structure crystal chemistry and bonding in rockforming minerals introduces the theory relating crystal structure and crystal symmetry to physical properties such as refractive index elastic modulus and seismic velocity surveys the distribution of silicate oxide and metallic minerals in the interiors and on the surfaces of planets and discusses the processes that led to their formation </code>	<code>this course provides a comprehensive introduction to crystalline structure crystal chemistry and bonding in rockforming minerals it introduces the theory relating crystal structure and crystal symmetry to physical properties such as refractive index elastic modulus and seismic velocity it surveys the distribution of silicate oxide and metallic minerals in the interiors and on the surfaces of planets and discusses the processes that led to their formation it also addresses why diamonds are hard and why micas split into thin sheets </code>	<code>0.949999988079071</code>
  <code>introduction to the theory of xray microanalysis through the electron microprobe including zaf matrix corrections techniques to be discussed are wavelength and energy dispersive spectrometry scanning backscattered electron secondary electron cathodoluminescence and xray imaging lab sessions involve the use of the electron microprobe the method is nondestructive and utilizes characteristic xrays excited by an electron beam incident on a flat surface of the sample lab sessions provide handson experience with the jeol jxa8200 superprobe</code>	<code>the electron microprobe provides a complete micrometerscale quantitative chemical analysis of inorganic solids the method is nondestructive and utilizes characteristic xrays excited by an electron beam incident on a flat surface of the sample this course provides an introduction to the theory of xray microanalysis through wavelength and energy dispersive spectrometry wds and eds zaf matrix correction procedures and scanning electron imaging with backscattered electron bse secondary electron se xray using wds or eds elemental mapping and cathodoluminescence cl lab sessions involve handson use of the jeol jxa8200 superprobe </code>	<code>0.949999988079071</code>

• Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 43 evaluation samples
• Columns: <code>description-mit</code>, <code>description-ocw</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	description-mit	description-ocw	label
  type	string	string	float
  details	<ul><li>min: 51 tokens</li><li>mean: 95.84 tokens</li><li>max: 150 tokens</li></ul>	<ul><li>min: 36 tokens</li><li>mean: 83.28 tokens</li><li>max: 175 tokens</li></ul>	<ul><li>min: 0.05</li><li>mean: 0.53</li><li>max: 0.95</li></ul>

• Samples:

  description-mit	description-ocw	label
  <code>presents basic principles of planet atmospheres and interiors applied to the study of extrasolar planets focuses on fundamental physical processes related to observable extrasolar planet properties provides a quantitative overview of detection techniques introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on extrasolar planets students taking graduate version complete additional assignments level of difficulty</code>	<code>this course covers the basic principles of planet atmospheres and interiors applied to the study of extrasolar planets exoplanets we focus on fundamental physical processes related to observable exoplanet properties we also provide a quantitative overview of detection techniques and an introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on exoplanets </code>	<code>0.6499999761581421</code>
  <code>presents basic principles of planet atmospheres and interiors applied to the study of extrasolar planets focuses on fundamental physical processes related to observable extrasolar planet properties provides a quantitative overview of detection techniques introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on extrasolar planets students taking graduate version complete additional assignments level of difficulty</code>	<code>this course covers the survey of the various subdisciplines of geophysics applied to the study of geodesy gravity geomagnetism seismology and geodynamics exoplanets we focus on fundamental physical processes related to observable exoplanet properties we also provide a quantitative overview of detection techniques and an introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on exoplanets </code>	<code>0.6499999761581421</code>
  <code>covers the basic concepts of sedimentation from the properties of individual grains to largescale basin analysis lectures cover sediment textures and composition fluid flow and sediment transport and formation of sedimentary structures depositional models for both modern and ancient environments are a major component and are studied in detail with an eye toward interpretation of depositional processes and reconstructing ecological dynamics from the rock record satisfies 6 units of institute laboratory credit level of difficulty students taking graduate version complete additional assignments students will explore siliciclastic and carbonate diagenesis and paleontology with a focus on fossils in sedimentary rocks</code>	<code>survey of the basic aspects of wave motion flow instability and turbulence emphasis is on fundamental materials features and processes textures of siliciclastic sediments and sedimentary rocks particle size particle shape and particle packing mechanics of sediment transport survey of the dynamics of surface and internal gravity waves poincare waves kelvin waves and topographic waves siliciclastic and carbonate diagenesis paleontology with special reference to fossils in sedimentary rocks modern and ancient depositional environments stratigraphy sedimentary basins fossil fuels coal petroleum covers 6 institute laboratory credit units</code>	<code>0.5</code>

• Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• num_train_epochs: 107
• warmup_ratio: 0.1
• fp16: True

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 107
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.0.1
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu118
• Accelerate: 0.30.0
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->