finetuned2-College-embeddings

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1 <!-- at revision 842eaed40bee4d61673a81c92d5689a8fed7a09f -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Bo8dady/finetuned2-College-embeddings")
# Run inference
sentences = [
    "Where can I find Abdel Badi Salem's email address?",
    'Dr. Abdel Badi Salem is part of the CS department and can be reached at [email protected].',
    "# **Abstract**\n\n## **Sports Analytics Overview**\nSports analytics has been successfully applied in sports like football and basketball. However, its application in soccer has been limited. Research in soccer analytics with Machine Learning techniques is limited and is mostly employed only for predictions. There is a need to find out if the application of Machine Learning can bring better and more insightful results in soccer analytics. In this thesis, we perform descriptive as well as predictive analysis of soccer matches and player performances.\n\n## **Football Rating Analysis**\nIn football, it is popular to rely on ratings by experts to assess a player's performance. However, the experts do not unravel the criteria they use for their rating. We attempt to identify the most important attributes of player's performance which determine the expert ratings. In this way we find the latent knowledge which the experts use to assign ratings to players. We performed a series of classifications with three different pruning strategies and an array of Machine Learning algorithms. The best results for predicting ratings using performance metrics had mean absolute error of 0.17. We obtained a list of most important performance metrics for each of the playing positions which approximates the attributes considered by the experts for assigning ratings. Then we find the most influential performance metrics of the players for determining the match outcome and we examine the extent to which the outcome is characterized by the performance attributes of the players. We found 34 performance attributes",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Dataset: ai-college-validation
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: ai-college-validation
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: ai-college-validation
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,030 training samples
• Columns: <code>Question</code> and <code>chunk</code>
• Approximate statistics based on the first 1000 samples:

  	Question	chunk
  type	string	string
  details	<ul><li>min: 8 tokens</li><li>mean: 15.99 tokens</li><li>max: 31 tokens</li></ul>	<ul><li>min: 21 tokens</li><li>mean: 133.41 tokens</li><li>max: 512 tokens</li></ul>

• Samples:

  Question	chunk
  <code>Could you share the link to the 2018 Distributed Computing final exam?</code>	<code>The final exam for Distributed Computing course, offered by the computer science department, from 2018, is available at the following link: [https://drive.google.com/file/d/1YSzMeYStlFEztP0TloIcBqnfPr60o4ez/view?usp=sharing</code>
  <code>What databases exist for footstep recognition research?</code>	<code>Abstract<br><br>Documentation Overview<br>This documentation reports an experimental analysis of footsteps as a biometric. The focus here is on information extracted from the time domain of signals collected from an array of piezoelectric sensors.<br><br>Database Information<br>Results are related to the largest footstep database collected to date, with almost 20,000 valid footstep signals and more than 120 persons, which is well beyond previous related databases.<br><br>Feature Extraction<br>Three feature approaches have been extracted, the popular ground reaction force (GRF), the spatial average and the upper and lower contours of the pressure signals.<br><br>Experimental Results<br>Experimental work is based on a verification mode with a holistic approach based on PCA and SVM, achieving results in the range of 5 to 15% equal error rate(EER) depending on the experimental conditions of quantity of data used in the reference models.</code>
  <code>Is there a maximum duration of study specified in the text?</code>	<code>Topic: Duration of Study<br>Summary: A bachelor's degree at the Faculty of Computers and Information requires at least four years of study, contingent on fulfilling degree requirements.<br>Chunk: "Duration of study<br>• The duration of study at the Faculty of Computers and Information to obtain a bachelor's degree is not less than 4 years, provided that the requirements for obtaining the scientific degree are completed."</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 575 evaluation samples
• Columns: <code>Question</code> and <code>chunk</code>
• Approximate statistics based on the first 575 samples:

  	Question	chunk
  type	string	string
  details	<ul><li>min: 9 tokens</li><li>mean: 15.97 tokens</li><li>max: 29 tokens</li></ul>	<ul><li>min: 21 tokens</li><li>mean: 134.83 tokens</li><li>max: 484 tokens</li></ul>

• Samples:

  Question	chunk
  <code>Are there projects that use machine learning for automatic brain tumor identification?</code>	<code># Abstract<br><br>## Brain and Tumor Description<br>A human brain is center of the nervous system; it is a collection of white mass of cells. A tumor of brain is collection of uncontrolled increasing of these cells abnormally found in different part of the brain namely Glial cells, neurons, lymphatic tissues, blood vessels, pituitary glands and other part of brain which lead to the cancer.<br><br>## Detection and Identification<br>Manually it is not so easily possible to detect and identify the tumor. Programming division method by MRI is way to detect and identify the tumor. In order to give precise output a strong segmentation method is needed. Brain tumor identification is really challenging task in early stages of life. But now it became advanced with various machine learning and deep learning algorithms. Now a day's issue of brain tumor automatic identification is of great interest. In Order to detect the brain tumor of a patient we consider the data of patients like MRI images of a pat...</code>
  <code>Are there studies that propose solutions to the challenges of plant pest detection using deep learning?</code>	<code>Abstract<br><br>Introduction<br>Identification of the plant diseases is the key to preventing the losses in the yield and quantity of the agricultural product. Disease diagnosis based on the detection of early symptoms is a usual threshold taken into account for integrated pest management strategies. through deep learning methodologies, plant diseases can be detected and diagnosed.<br><br>Study Discussion<br>On this basis, this study discusses possible challenges in practical applications of plant diseases and pests detection based on deep learning. In addition, possible solutions and research ideas are proposed for the challenges, and several suggestions are given. Finally, this study gives the analysis and prospect of the future trend of plant diseases and pests detection based on deep learning.<br><br>5
  <code>Is there a link available for the 2025 Calc 1 course exam?</code>	<code>The final exam for the calculus1 course, offered by the general department, from 2025, is available at the following link: [https://drive.google.com/file/d/1g8iiGUo4HCUzNNWBJJrW1QZAsz-RYehw/view?usp=sharing].</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 1e-06
• warmup_ratio: 0.2
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-06
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.2
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.51.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.5.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->