marginmse_experiment_gte

SentenceTransformer based on PaDaS-Lab/xlm-roberta-base-msmarco

This is a sentence-transformers model finetuned from PaDaS-Lab/xlm-roberta-base-msmarco. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: PaDaS-Lab/xlm-roberta-base-msmarco <!-- at revision cd02f4c38b71baa0dc6b3fcdd86a3b6bd407ef55 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Impact of the physico-chemical properties of fen peat on the metal accumulation patterns in mires of Latvia',
    'Santrauka\nAbstract The article presents a study of the physico-chemical properties of fen peat and their influence on the metal accumulation patterns in three Latvian fens: Svētupes Mire, Elku Mire and Vīķu Mire. Full peat profiles were obtained at all study sites and analysed with a multi-proxy approach. The content of metals in fen peat was determined using the atomic absorption spectroscopy (AAS) and normalised to the concentration of Ti in the studied peat profiles. Both the character of deposits and agricultural land use in the mire catchment areas were taken into account and the possible natural and anthropogenic metal supply sources were evaluated. The content of metals in the studied fen peat significantly varied due to the heterogeneity of fen environment; however, noticeable similarities were also traced throughout all study sites. The results indicate an increased amount of transition metals and Pb in the upper peat layer. This can be explained by a direct impact from anthropogenic sources (agricultural land use, pollution, etc.). Metal binding in fen peat profiles is directly related to the alkali and alkaline earth metal content in peat, as Ca, Mg, Na and K ions are replaced by more tightly bound metal ions. In raised bogs, in turn, metal binding is associated with the acidic functional groups common to peat.\nDoi\xa010.5200/baltica.2016.29.03Raktažodžiai\xa0fen peat, metals, peat physico-chemical properties\nPilnas tekstas',
    'Wyckoff Method identifies accumulation and distribution phases by analyzing price and volume patterns. The goal is to locate support and resistance levels and patterns such as Springs and Upthrusts. Volume trends and moving averages provide confirmation.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9981, 0.9960],
#         [0.9981, 1.0000, 0.9978],
#         [0.9960, 0.9978, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 12,753,278 training samples
• Columns: <code>sentence_0</code>, <code>sentence_1</code>, <code>sentence_2</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2	label
  type	string	string	string	float
  details	<ul><li>min: 6 tokens</li><li>mean: 15.49 tokens</li><li>max: 119 tokens</li></ul>	<ul><li>min: 9 tokens</li><li>mean: 74.63 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: 11 tokens</li><li>mean: 107.12 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: -0.48</li><li>mean: 0.16</li><li>max: 0.82</li></ul>

• Samples:

  sentence_0	sentence_1	sentence_2	label
  <code>Как найти актуальное зеркало Betwinner?</code>	<code>Букмекер старается обеспечить доступ к сайту, поэтому ссылки на зеркала обновляются ежедневно. Чтобы быть в курсе всех новостей, рекомендуется подписаться на почтовую рассылку и соцсети.</code>	<code>Актуальные рабочие зеркала 1win можно найти на официальных страницах букмекерской конторы в социальных сетях или путем обращения в службу поддержки.</code>	<code>-0.253662109375</code>
  <code>Jakie są minimalne zakłady w bakaracie?</code>	<code>Minimalny zakład w bakaracie zależy od konkretnej gry, w którą grasz. Mini Baccarat ma zwykle niskie limity zakładów, co czyni go atrakcyjnym dla nowych graczy. Istnieją też wersje gry w bakarata dla high-rollerów, które nakładają wyższy minimalny zakład.</code>	<code>W czasie pisania tego tekstu, Cloudbet miał minimalny zakład w wysokości 0,01 BTC. Cloudbet ma jedne z najwyższych dostępnych maksymalnych zakładów, ale zazwyczaj różnią się one w zależności od płynności na dane zdarzenie. Ogólnie rzecz biorąc, im bliższe zdarzeniu zakłady, tym większa płynność i tym wyższe stawki można postawić.</code>	<code>0.3369140625</code>
  <code>Come scegliere il massimale assicurazione professionale medici?</code>	<code>Per il momento, non sono ancora entrate in vigore sul massimale minimo per le polizze rc professionale medici. Teniamo conto però di una cosa: se si lavora (e si è lavorato nei dieci anni precedenti) esclusivamente come dipendenti o specializzandi presso l’SSN, dobbiamo sapere che la rivalsa massima dell’SSN sarà plafonata al triplo del reddito annuo lordo del medico.<br>Se invece si lavora in libera professione, non c’è alcun limite. Consigliamo comunque di scegliere massimali non inferiori al milione di euro.</code>	<code>Potete stipulare una nuova assicurazione di base entro il 31 dicembre, che inizierà a decorrere dal 1 gennaio dell’anno successivo. A tal fine, però, dovete aver disdetto la precedente assicurazione di base entro i termini previsti.<br>In linea di massima è possibile disdire l’assicurazione di base inviando una lettera di disdetta firmata per posta, via e-mail o via fax. Alla KPT potete comunicare la disdetta anche nel portale clienti KPTnet.<br>Attenzione: Per la disdetta non fa fede il timbro postale, bensì la data in cui l’assicuratore riceve la disdetta. Il termine di disdetta è rispettato se l’assicuratore riceve la disdetta l’ultimo giorno lavorativo del termine legale durante i normali orari di ufficio. Un invio raccomandato recapitato in una casella postale può essere considerato notificato solo nel momento in cui è ritirato alla Posta. Poiché non si possono escludere ritardi, vi consigliamo di inviare la disdetta per posta, mediante lettera raccomandata, entro la metà di novembre.<br>C...</code>	<code>0.09228515625</code>

• Loss: <code>MarginMSELoss</code>

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• num_train_epochs: 1
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.10.4
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 2.21.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->