all-MiniLM-L6-v83-pair_score

all-MiniLM-L6-v83-pair_score

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the pairs_with_scores_v67 dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 -->
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • pairs_with_scores_v67
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'henge chair and bookcase 2',
    'neck free bandana - raindropgrey libra bandana ultragrip bandana workout bandana nonslip bandana velvet bandana bandana neck free bandana raindrop bandana bandana neck free bandana raindrop bandana',
    'bali floor lamp handpick handpick floor lamp rustic floor lamp palm leaves floor lamp black floor lamp metal base floor lamp bali lamp floor lamp lamp bali lamp floor lamp lamp',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -0.1181,  0.0299],
#         [-0.1181,  1.0000, -0.1149],
#         [ 0.0299, -0.1149,  1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

pairs_with_scores_v67

• Dataset: pairs_with_scores_v67 at 38f07b3
• Size: 65,396,625 training samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	<ul><li>min: 3 tokens</li><li>mean: 7.91 tokens</li><li>max: 27 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 45.72 tokens</li><li>max: 256 tokens</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.01</li><li>max: 1.0</li></ul>

• Samples:

  sentence1	sentence2	score
  <code>baladi qeshta plate</code>	<code>optimum nutrition glutamine 300g powder 58 servings ws nutrition glutamine powder glutamine powder boost protein synthesis powder optimum nutrition powder enhance workout performance powder improve muscle recovery powder glutamine glutamine powder muscle builder powder optimum nutrition optimum nutrition glutamine powder optimum nutrition powder glutamine powder muscle builder powder optimum nutrition optimum nutrition glutamine powder optimum nutrition powder</code>	<code>0.0</code>
  <code>diabetic pecan bar</code>	<code>round rose shape cake mold nonstick cake mold healthy cooking cake mold whitford cake mold heatinsulated handles cake mold stainless steel cover cake mold pfoafree cake mold rose cake mold cookware cake mold round cake mold cake mold round cake mold</code>	<code>0.0</code>
  <code>red velvet cupcakes</code>	<code>total plastic fencing shears 22 tools equipment fencing shears plastic shears shears total plastic fencing shears fencing shears plastic shears shears total plastic fencing shears</code>	<code>0.0</code>

• Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

pairs_with_scores_v67

• Dataset: pairs_with_scores_v67 at 38f07b3
• Size: 328,627 evaluation samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	<ul><li>min: 3 tokens</li><li>mean: 7.8 tokens</li><li>max: 28 tokens</li></ul>	<ul><li>min: 5 tokens</li><li>mean: 44.87 tokens</li><li>max: 256 tokens</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.01</li><li>max: 1.0</li></ul>

• Samples:

  sentence1	sentence2	score
  <code>mint hot chocolate</code>	<code>baba ghanouj dip v baba ghanouj dip dip vegan baba ghanouj dip baba ghanouj dip dip vegan baba ghanouj dip</code>	<code>0.0</code>
  <code>sponge balls gun toy</code>	<code>multi color rug colorful rugs summer house rugs beach mat multi color rug rug carpet multi color carpet multi nan rug multi color rug rug carpet multi color carpet multi nan rug</code>	<code>0.0</code>
  <code>the knit jumpsuit</code>	<code>spinaci alla paradiso spinach pasta mushroom fettuccine red bell pepper pasta garlic fettuccine creamy fettuccine parmigiano pasta olive oil fettuccine parmesan garlic fettuccine vegetarian pasta fettuccine italian pasta pasta spinaci alla paradiso pasta italian macarona macarona spinaci alla paradiso macarona italian pasta pasta spinaci alla paradiso pasta italian macarona macarona spinaci alla paradiso macarona</code>	<code>0.0</code>

• Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.1.0
• Transformers: 4.55.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->