Sentence-Transformer_Roberta_Reddit_ed

SentenceTransformer

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    ' still has a lot of food fears, incredibly picky regardless and lots of food anxiety\\* this is important',
    ' My relationship with food was much different in high school, I would eat without even thinking about it',
    " I've been trying to keep down breakfast and supper everyday plus some snacks and I don't think it's enough but I don't know if I'd be able to recover if I was eating any more than that",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6203, 0.5744],
#         [0.6203, 1.0000, 0.7722],
#         [0.5744, 0.7722, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Binary Classification

• Dataset: quora_duplicates_dev
• Evaluated with <code>BinaryClassificationEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,000 training samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	int
  details	<ul><li>min: 4 tokens</li><li>mean: 33.52 tokens</li><li>max: 169 tokens</li></ul>	<ul><li>min: 5 tokens</li><li>mean: 32.67 tokens</li><li>max: 228 tokens</li></ul>	<ul><li>0: ~57.10%</li><li>1: ~42.90%</li></ul>

• Samples:

  sentence1	sentence2	label
  <code> Obviously as my age is specified there's not much I can do from a physical standpoint other than cook for her when I'm with her, but I'e been there for her emotionally in the past for other things and I think it helped her</code>	<code> any advice would be greatly appreciated to help her</code>	<code>1</code>
  <code> I am 15 years old and recently I have been feeling awful about it and realized how much of a problem it has become</code>	<code> <br><br>And I know the answer is probably, "go see a doctor or therapist/talk to ur parents" however I'm worried about doing that since what if it's nothing but me worrying and making a deal about nothing, I don't want them judging me over something potentially stupid like this</code>	<code>0</code>
  <code> This is Part 1 of 2 for 1 post) A friend asked me for some advice for her sister who is currently struggling with her relationship to food and her body</code>	<code> <br><br>The things I'm concerned about:<br>- I don't like eating, I try to avoid it until my stomach physically hurts, especially when I have a major depressive episode</code>	<code>1</code>

• Loss: <code>ContrastiveLoss</code> with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 1.0,
      "size_average": true
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 760 evaluation samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
• Approximate statistics based on the first 760 samples:

  	sentence1	sentence2	label
  type	string	string	int
  details	<ul><li>min: 4 tokens</li><li>mean: 33.98 tokens</li><li>max: 230 tokens</li></ul>	<ul><li>min: 4 tokens</li><li>mean: 31.82 tokens</li><li>max: 192 tokens</li></ul>	<ul><li>0: ~58.55%</li><li>1: ~41.45%</li></ul>

• Samples:

  sentence1	sentence2	label
  <code> I'm not getting enough nutrients ?<br><br>I have a lot of unhealthy coping mechanisms and I'm trying to not SH as much anymore, which makes that the ED is coming up much stronger? <br>I don't know what to do or how to stop this, anyone have any advice?<br><br>Edit: I also have to say that when I eat I get really nauseous, but when I don't eat I also get nauseous, which then makes it harder to eat and it just keeps going on like that </code>	<code> I want to be smaller than them and all that because eds are like that, but that's hard when they don't tell me stuff and I'm left to stew in my own thoughts</code>	<code>0</code>
  <code> I'm underweight right now and I was actually gaining weight since I was an average weight about two-three weeks ago and went down</code>	<code> So, this is definetly going into the right direction, but:<br> I am so afraid of stepping on the scale and seeing gained weight</code>	<code>1</code>
  <code> I feel so much shame around the idea of eating junk food and often times, when things have gotten worse, part of that mechanism has been my definition of "junk food" getting so warped that I consider almost anything unhealthy and shameful to eat</code>	<code> Overall, I realized my eating disorders had almost nothing to do with food</code>	<code>0</code>

• Loss: <code>ContrastiveLoss</code> with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 1.0,
      "size_average": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 2
• learning_rate: 1.5e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1.5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.10.8
• Sentence Transformers: 5.0.0
• Transformers: 4.49.0
• PyTorch: 2.1.2+cu118
• Accelerate: 1.4.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->