MNLP_M3_tokenizer_tuned

SentenceTransformer based on sucharush/e5_stem_finetuned

This is a sentence-transformers model finetuned from sucharush/e5_stem_finetuned. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sucharush/e5_stem_finetuned <!-- at revision 3013cc95b53a83082746afd556743e81f1da5dff -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Stergios-Konstantinidis/MNLP_M3_tokenizer_tuned")
# Run inference
sentences = [
    '[{\'from\': \'human\', \'value\': \'Solve the following math problem step-by-step.\\nSimplify your answer as much as possible. Present your final answer as \\\\boxed{Your Answer}.\\nA pentagon is drawn by placing an isosceles right triangle on top of a square as pictured. What percent of the area of the pentagon is the area of the right triangle?\\n\\n[asy]\\nsize(50);\\ndraw((0,0)--(0,-1)--(1,-1)--(1,0)--(0,0)--(.5,.5)--(1,0));\\n[/asy]\'}, {\'from\': \'gpt\', \'value\': "Step 1:  We are given a pentagon formed by placing an isosceles right triangle on top of a square.\\nStep 2:  To find the percentage of the area of the pentagon that is the right triangle, we need to consider the areas of both shapes and then calculate the ratio of the triangle\'s area to the total pentagon\'s area.\\nStep 3:  Let\'s designate the leg length of the isosceles right triangle as $x$. This means the hypotenuse of the triangle has a length of $x\\\\sqrt{2}$.\\nStep 4:  Since the hypotenuse of the triangle is a side of the square, the area of the square is $(x\\\\sqrt{2})^2 = 2x^2$. \\nStep 5:  On the other hand, the area of the triangle is $(x)(x)/2 = x^2/2$. This is obtained using the standard formula for the area of a triangle, which is base times height divided by 2.\\nStep 6:  So, the area of the pentagon is the sum of the areas of the triangle and the square, which gives us $\\\\frac{x^2}{2} + 2x^2 = \\\\frac{5x^2}{2}$.\\nStep 7:  Now, to find the percentage of the pentagon\'s area that is inside the triangle, we calculate the ratio of the triangle\'s area to the total pentagon\'s area. This can be computed as $\\\\frac{x^2/2}{5x^2/2} =\\\\frac{x^2}{2}\\\\cdot \\\\frac{2}{5x^2} = \\\\frac15$.\\nStep 8:  Therefore, the percentage of the area of the pentagon that is the right triangle is $\\\\frac{1}{5}$, which is equivalent to $\\\\boxed{20\\\\%}$.\\nStep 9:  As an alternate solution, we can consider drawing the two diagonals of the square and observe the resulting geometric properties to arrive at the same conclusion.\\n\\nAnswer:\\n\\\\boxed{20\\\\%}"}]',
    '[{\'from\': \'human\', \'value\': \'Solve the following math problem step-by-step.\\nSimplify your answer as much as possible. Present your final answer as \\\\boxed{Your Answer}.\\nA pentagon is drawn by placing an isosceles right triangle on top of a square as pictured. What percent of the area of the pentagon is the area of the right triangle?\\n\\n[asy]\\nsize(50);\\ndraw((0,0)--(0,-1)--(1,-1)--(1,0)--(0,0)--(.5,.5)--(1,0));\\n[/asy]\'}, {\'from\': \'gpt\', \'value\': "Step 1:  We are given a pentagon formed by placing an isosceles right triangle on top of a square.\\nStep 2:  To find the percentage of the area of the pentagon that is the right triangle, we need to consider the areas of both shapes and then calculate the ratio of the triangle\'s area to the total pentagon\'s area.\\nStep 3:  Let\'s designate the leg length of the isosceles right triangle as $x$. This means the hypotenuse of the triangle has a length of $x\\\\sqrt{2}$.\\nStep 4:  Since the hypotenuse of the triangle is a side of the square, the area of the square is $(x\\\\sqrt{2})^2 = 2x^2$. \\nStep 5:  On the other hand, the area of the triangle is $(x)(x)/2 = x^2/2$. This is obtained using the standard formula for the area of a triangle, which is base times height divided by 2.\\nStep 6:  So, the area of the pentagon is the sum of the areas of the triangle and the square, which gives us $\\\\frac{x^2}{2} + 2x^2 = \\\\frac{5x^2}{2}$.\\nStep 7:  Now, to find the percentage of the pentagon\'s area that is inside the triangle, we calculate the ratio of the triangle\'s area to the total pentagon\'s area. This can be computed as $\\\\frac{x^2/2}{5x^2/2} =\\\\frac{x^2}{2}\\\\cdot \\\\frac{2}{5x^2} = \\\\frac15$.\\nStep 8:  Therefore, the percentage of the area of the pentagon that is the right triangle is $\\\\frac{1}{5}$, which is equivalent to $\\\\boxed{20\\\\%}$.\\nStep 9:  As an alternate solution, we can consider drawing the two diagonals of the square and observe the resulting geometric properties to arrive at the same conclusion.\\n\\nAnswer:\\n\\\\boxed{20\\\\%}"}]',
    'Question: where are the basilar membrane and the organ of corti located, Answer: Organ of Corti The organ of Corti is located in the scala media of the cochlea of the inner ear between the vestibular duct and the tympanic duct and is composed of mechanosensory cells, known as hair cells.[2] Strategically positioned on the basilar membrane of the organ of Corti are three rows of outer hair cells (OHCs) and one row of inner hair cells (IHCs).[4] Separating these hair cells are supporting cells: Deiters cells, also called phalangeal cells, which separate and support both the OHCs and the IHCs.[4]',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 113,450 training samples
• Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code>
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	int
  details	<ul><li>min: 17 tokens</li><li>mean: 283.42 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: 16 tokens</li><li>mean: 281.46 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>0: ~80.00%</li><li>1: ~20.00%</li></ul>

• Samples:

  sentence_0	sentence_1	label
  <code>Question: where does rasin in the sun take place, Answer: A Raisin in the Sun A Raisin in the Sun is a play by Lorraine Hansberry that debuted on Broadway in 1959.[1] The title comes from the poem "Harlem" (also known as "A Dream Deferred"[2]) by Langston Hughes. The story tells of a black family's experiences in the Washington Park Subdivision of Chicago's Woodlawn neighborhood as they attempt to "better" themselves with an insurance payout following the death of the father. The New York Drama Critics' Circle named it the best play of 1959.</code>	<code>Question: where does rasin in the sun take place, Answer: A Raisin in the Sun A Raisin in the Sun is a play by Lorraine Hansberry that debuted on Broadway in 1959.[1] The title comes from the poem "Harlem" (also known as "A Dream Deferred"[2]) by Langston Hughes. The story tells of a black family's experiences in the Washington Park Subdivision of Chicago's Woodlawn neighborhood as they attempt to "better" themselves with an insurance payout following the death of the father. The New York Drama Critics' Circle named it the best play of 1959.</code>	<code>1</code>
  <code>Question: when does the movie midnight sun come out, Answer: Midnight Sun (2018 film) Midnight Sun is a 2018 American romantic drama film directed by Scott Speer and written by Eric Kirsten, based on the 2006 Japanese film of the same name. The film stars Bella Thorne, Patrick Schwarzenegger, and Rob Riggle, and follows a teenage girl with the disease xeroderma pigmentosum, which prevents her from going out into sunlight. When she meets a boy, she struggles to decide whether to tell him about her condition or pretend to live a normal life. Principal photography began on October 12, 2015 in Vancouver, British Columbia, Canada. The film was released in the United States on March 23, 2018.</code>	<code>Question: who sang the song unchained melody in the movie ghost, Answer: Unchained Melody "Unchained Melody" reappeared on the Billboard charts in 1990 after The Righteous Brothers' recording was used in the box office blockbuster film Ghost. Two versions charted in the US that year – the original and a new recording. According to Medley, he was interested in having the original recording released due to the renewed interest in the song, but was told that there were licensing issues. Although Hatfield was no longer in the same condition vocally as when he first recorded the song, they decided to re-record the song for Curb Records.[44] The re-recorded version was released as both a cassette single[45] and a CD single.[46] It received minimal airplay but recorded excellent sales, peaking at number 19. The re-recorded version was certified Platinum by the RIAA on January 10, 1991,[47] and received a Grammy Award nomination.[48]</code>	<code>0</code>
  <code>Question: where is the o2 arena in london located, Answer: The O2 Arena The O2 Arena (temporarily the sponsor neutral "North Greenwich Arena", during the 2012 Summer Olympics and 2012 Summer Paralympics), is a multi purpose indoor arena located in the centre of The O2 entertainment complex on the Greenwich Peninsula in south east London.</code>	<code>[{'from': 'human', 'value': "Explain how Archimedes' Principle enables objects to float in water."}, {'from': 'gpt', 'value': "Archimedes' Principle states that the buoyant force acting on an object submerged in water is equal to the weight of the water displaced by the object. If the buoyant force exceeds the weight of the object, it will float."}]</code>	<code>0</code>

• Loss: <code>ContrastiveTensionLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 3.4.1
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu126
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveTensionLoss

@inproceedings{carlsson2021semantic,
    title={Semantic Re-tuning with Contrastive Tension},
    author={Fredrik Carlsson and Amaru Cuba Gyllensten and Evangelia Gogoulou and Erik Ylip{"a}{"a} Hellqvist and Magnus Sahlgren},
    booktitle={International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=Ov_sMNau-PF}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->