deep-learning-for-embedding-model-ssilwal-qpham6

deep learning project

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5 <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bbmb/deep-learning-for-embedding-model-ssilwal-qpham6")
# Run inference
sentences = [
    "9. Area\n\nEquation 9.35 calculates the area, ( A ), bounded by ( x = a ), ( x = b ), ( f_1(x) ) above, and ( f_2(x) ) below. (Note: ( f_2(x) = 0 ) if the area is bounded by the x-axis.) This is illustrated in Fig. 9.1. [ A = \\int_{a}^{b} \\left( f_{1}(x) - f_{2}(x) \\right) \\, dx \\qquad \\qquad (9.35) ] Figure 9.1 Area Between Two Curves\n\nDescription: The image shows a graph with two curves labeled f1(x) and f2(x). The graph is plotted on a Cartesian coordinate system with an x-axis and a y-axis. There are two vertical dashed lines intersecting the x-axis at points labeled 'a' and 'b'. The curve f1(x) is above the line y = 0 and the curve f2(x) is below the line y = 0. The area between the two curves from x = a to x = b is shaded, indicating a region of interest or calculation.\n\nThe LaTeX representation of the curves is not provided in the image, so I cannot write them in LaTeX form. However, if the curves were described by functions, they could be represented as follows:\n\nf1(x) could be represented as ( f_1(x) = ax^2 + bx + c ) for some constants a, b, and c.\n\nf2(x) could be represented as ( f_2(x) = -ax^2 - bx - c ) for some constants a, b, and c.\n\nThe area between the curves from x = a to x = b could be calculated using the integral of the difference between the two functions over the interval [a, b].\n\nDescription: The image provided is not clear enough to describe in detail or to extract any formulas. The text is not legible, and no other discernible features can be identified.\n\nFind the area between the x-axis and the parabola ( y = x^2 ) in the interval ([0, 4]).\n\nDescription: The image shows a graph with a curve that represents a function y = x^2. There is a vertical dashed line at x = 4, indicating a point of interest or a specific value on the x-axis. The graph is plotted on a Cartesian coordinate system with the x-axis labeled 'x' and the y-axis labeled 'y'. The curve is a parabola that opens upwards, showing that as x increases, y increases at an increasing rate. The point where x = 4 is marked on the x-axis, and the corresponding y-value on the curve is not explicitly shown but can be inferred from the equation y = x^2.\n\nSolution: Referring to Eq. 9.35, [ f_{1}(x) = x^{2} \\quad \\text{and} \\quad f_{2}(x) = 0 ] Thus, [ A = \\int_{a}^{b} \\left( f_1(x) - f_2(x) \\right) dx = \\int_{0}^{4} x^2 \\, dx = \\left[ \\frac{x^3}{3} \\right]_{0}^{4} = \\frac{64}{3} ] ...\n\n10. Arc Length",
    'Can you show me how to find the area using the integral of the difference of two functions?',
    'What is the minimum requirement for steel area in slab reinforcement according to ACI guidelines?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

json

• Dataset: json
• Size: 3,078 training samples
• Columns: <code>positive</code> and <code>anchor</code>
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	<ul><li>min: 117 tokens</li><li>mean: 508.1 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: 8 tokens</li><li>mean: 15.93 tokens</li><li>max: 28 tokens</li></ul>

• Samples:

  positive	anchor
  <code>The PHF is used to convert hourly volumes to flow rates and represents the hourly variation in traffic flow. If the demand volume is measured in 15 min increments, it is unnecessary to use the PHF to convert to flow rates.<br><br>Therefore, since two-lane highway analysis is based on demand flow rates for a peak 15 min period within the analysis hour (usually the peak hour), the PHF in Equation 73.22 and Equation 73.23 is given a value of 1.00.<br><br>The average travel speed in the analysis direction, ( ATS_d ), is estimated from the FFS, the demand flow rate, the opposing flow rate, and the adjustment factor for the percentage of no-passing zones in the analysis direction, ( f_{np} ), as given in HCM Exh. 15-15. Equation 73.24 only applies to Class I and Class III two-lane highways.<br><br>[ \mathrm{ATS}{d} = \mathrm{FFS} - 0.0076(v{d,s} + v_{o,s}) - f_{\mathrm{np},s} \quad (73.24) ]<br><br>If the PTSF methodology is used, the formula for the demand flow rate, ( v_{i, \text{ATS}} ), is the same, although di...</code>	<code>What is the formula for estimating the percent time spent following on highways?</code>
  <code>However, if the initial point on the limb is close to the critical point (i.e., the nose of the curve), then a small change in the specific energy (such as might be caused by a small variation in the channel floor) will cause a large change in depth. That is why severe turbulence commonly occurs near points of critical flow. Given that ( 4 , \text{ft/sec} ) (or ( 1.2 , \text{m/s} )) of water flows in a ( 7 , \text{ft} ) (or ( 2.1 , \text{m} )) wide, ( 6 , \text{ft} ) (or ( 1.8 , \text{m} )) deep open channel, the flow encounters a ( 1.0 , \text{ft} ) (or ( 0.3 , \text{m} )) step in the channel bottom. What is the depth of flow above the step? Actually, specific energy curves are typically plotted for flow per unit width, ( q = \frac{Q}{w} ). If that is the case, a jump from one limb to the other could take place if the width were allowed to change as well as the depth. A rise in the channel bottom does not always produce a drop in the water surface. Only if the flow is initiall...</code>	<code>What happens to the water depth when it encounters a step in a channel?</code>
  <code>The shear strength, ( S ) or ( S_{ys} ), of a material is the maximum shear stress that the material can support without yielding in shear. (The ultimate shear strength, ( S_{us} ), is rarely encountered.) For ductile materials, maximum shear stress theory predicts the shear strength as one-half of the tensile yield strength. A more accurate relationship is derived from the distortion energy theory (also known as von Mises theory).<br><br>Figure 43.16: Uniform Bar in Torsion<br><br>Description: The image shows a diagram of a mechanical system with a cylindrical object, a rod, and a spring. There are two forces acting on the system: one is the weight of the rod, labeled 'L', acting downwards, and the other is the spring force, labeled 'T', acting upwards. The rod is shown to be in equilibrium, with the spring force balancing the weight of the rod. The distance from the pivot point to the center of mass of the rod is labeled 'r'. There is also a variable 'y' indicating the vertical displacement of t...</code>	<code>Can you explain what maximum shear stress theory is?</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.34.2
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->