all-MiniLM-L6-v2-LoRA-finetuned-1M

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: PeftModelForFeatureExtraction 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("manupande21/all-MiniLM-L6-v2-LoRA-finetuned-1M")
# Run inference
sentences = [
    'what is the purpose of bands with braces',
    'The Purpose of Rubber Bands for Braces. If you have gotten a professional consultation from an orthodontist, you may have been given rubber bands for braces; lots of doctors recommend using rubber bands in conjunction with braces to help straighten teeth quicker.',
    'To successfully complete your orthodontic treatment plan, patients must work together with the orthodontist. The teeth and jaws can only move toward their corrected positions if the patient consistently wears the elastics (rubber bands), headgear or other appliances as prescribed. The following paragraphs describe the types of appliances that may be used during your treatment. Elastics (Rubber Bands) Wearing elastics (rubber bands) is the main way that Orthodontic treatment can improve the fit of your bite. Braces will straighten your teeth, but are unable to correct bite problems unless used in conjunction with elastics.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Triplet

• Dataset: test-eval
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 800,000 training samples
• Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>sentence_2</code>
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	<ul><li>min: 4 tokens</li><li>mean: 9.05 tokens</li><li>max: 38 tokens</li></ul>	<ul><li>min: 15 tokens</li><li>mean: 80.09 tokens</li><li>max: 232 tokens</li></ul>	<ul><li>min: 21 tokens</li><li>mean: 78.11 tokens</li><li>max: 192 tokens</li></ul>

• Samples:

  sentence_0	sentence_1	sentence_2
  <code>american modernism is characterized by _____.</code>	<code>FROM MODERNISM TO POST-MODERNISM. POST-WAR ART IN AMERICA. After the Second World War, the art world was characterized by “triumphalism” in New York and a feeling of having won, not just a military war but also a cultural war. The French and their School of Paris had been routed. Also defeated was American Scene painting and its nativist illustrations of a naïve nation.</code>	<code>noun. 1 a disease of sheep characterized by an unsteady gait and staggering, caused by infestation of the brain with tapeworms (Taenia caenuris).oun. 1 a disease of sheep characterized by an unsteady gait and staggering, caused by infestation of the brain with tapeworms (Taenia caenuris).</code>
  <code>where was sen tom cotton born</code>	<code>Written By: born. Tom Cotton, in full Thomas Bryant Cotton (born May 13, 1977, Dardanelle, Arkansas, U.S.), American politician who was elected to the U.S. Senate as a Republican in 2014 and began his first term representing Arkansas the following year. He previously was a member of the U.S. House of Representatives (2013–15).</code>	<code>Cotton Facts: 1 The Cotton Belt spans the southern half of the United States, stretching from Virginia to California. 2 Cotton production covers more than 14 million acres or about 22,000 square miles of the United States.3 Texas is the leading cotton-producing state, producing about 4.5 million bales of cotton a year.4 Cotton contributes over $1 5 ... Tex Cotton production covers more than 14 million acres or about 22,000 square miles of the United States. 2 Texas is the leading cotton-producing state, producing about 4.5 million bales of cotton a year. 3 Cotton contributes over $1 billion to the Texas economy, ranking only behind the beef industry in total cash receipts.</code>
  <code>what is ginger beer</code>	<code>Gosling's Ginger Beer. Crabbie’s Alcoholic Ginger Beer was first crafted in 1801. Ginger beer is carbonated, sweetened beverage produced in two versions: alcoholic brewed ginger beer (which includes home-brewed) or a carbonated soft drink flavored primarily with ginger and sweetened with sugar or artificial sweeteners.oday, ginger beer is usually produced as a soft drink. Ginger beer and ginger ale as soft drinks have been moderately popular in many parts of the world since they were introduced. The similarities and differences between ginger ale and ginger beer are discussed in the history section of the ginger ale article.</code>	<code>Ginger is a commonly used flavoring agent and food product. Ginger is also available as an herbal supplement. The information contained in this leaflet refers to the use of ginger as an herbal supplement.</code>

• Loss: <code>TripletLoss</code> with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

</details>

Training Logs

Framework Versions

• Python: 3.11.5
• Sentence Transformers: 4.1.0
• Transformers: 4.41.0
• PyTorch: 2.7.0+cu126
• Accelerate: 1.7.0
• Datasets: 3.2.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->