legal-ft-demo_final

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-m <!-- at revision fc74610d18462d218e312aa986ec5c8a75a98152 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bsmith3715/legal-ft-demo_final")
# Run inference
sentences = [
    'What muscle groups are primarily engaged during the lunge exercise described in the context?',
    "hips and we're gonna lunge it down the\nweight is going to feel a little bit\nlight we're working more stabilizers to\nstart here\nstabilizers through the hips ankles\nknees\nall the things okay lunge it down I like\nto put my foot right up against that\nedge\nto help me have a nice grip toes are off\nof the carriage\nand then coming up to squeeze up on a\nstraight standing leg squeeze up through\nthe glute\ndown\nand\nsqueeze and lift good\nthe slower you move here the more work\nyou're going to feel through those quads\nand glutes as well\nslow back\nslow up\nI know sometimes we feel like we want to\nget that heart rate going\nbut sometimes we need this slow movement\nis going to give us even more benefits\nto support us for those fast movements\nlater",
    "one\nfind that lengthened position you're\ngoing to lift up slide those shoulder\nblades down the back lift up towards the\nsky big inhale\nand exhale back and away\nleft hand to Center\nturn back towards me and lift I\napologize if you're not on the same side\nwhen you're facing me\nother arm to Center bottom arm all right\nwe're gonna flip around\nto do the same thing on the other side\nso lifting up tall\nplace that hand in front of your\nshoulder we're going up and over long\nspine and then lift shoulders down again\nup and over\nand then you use that oblique to lift up\nexhale lift\nand lengthen\nopen the spine Flex\noblique to come up\nand three\ntwo\nand one\ngood full mermaid now up and over turn\nto face the ground separate those arms",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 200 training samples
• Columns: <code>sentence_0</code> and <code>sentence_1</code>
• Approximate statistics based on the first 200 samples:

  	sentence_0	sentence_1
  type	string	string
  details	<ul><li>min: 12 tokens</li><li>mean: 16.91 tokens</li><li>max: 24 tokens</li></ul>	<ul><li>min: 96 tokens</li><li>mean: 159.61 tokens</li><li>max: 176 tokens</li></ul>

• Samples:

  sentence_0	sentence_1
  <code>What type of spring is the instructor using for the workout?</code>	<code>hi guys thanks for joining me today we<br>have a really fun really challenging<br>workout for you today before we get<br>started don't forget to like share<br>subscribe feel free to leave me those<br>super likes I really appreciate you guys<br>joining me for these workouts we're<br>going to get started setting up foot<br>bars all the way down I'm gonna go on to<br>one red Spring today which is going to<br>be one heavy spring on my reformer again<br>I'm gonna go really heavy for my arms<br>today if this is way too much for you<br>guys you can do a blue instead of a red<br>or a medium instead of a heavy spring<br>again it is going to be very heavy for<br>arms so feel free to change as needed we<br>are going to start first by straddling<br>your reformers your feet are on the</code>
  <code>What should participants do if the red spring is too heavy for them?</code>	<code>hi guys thanks for joining me today we<br>have a really fun really challenging<br>workout for you today before we get<br>started don't forget to like share<br>subscribe feel free to leave me those<br>super likes I really appreciate you guys<br>joining me for these workouts we're<br>going to get started setting up foot<br>bars all the way down I'm gonna go on to<br>one red Spring today which is going to<br>be one heavy spring on my reformer again<br>I'm gonna go really heavy for my arms<br>today if this is way too much for you<br>guys you can do a blue instead of a red<br>or a medium instead of a heavy spring<br>again it is going to be very heavy for<br>arms so feel free to change as needed we<br>are going to start first by straddling<br>your reformers your feet are on the</code>
  <code>What is the initial position described for starting the workout on the reformers?</code>	<code>are going to start first by straddling<br>your reformers your feet are on the<br>floor we're going to take our hands to<br>our shoulder blocks we're just going to<br>do a quick stretch before we get moving<br>so I'm going to bend my knees slightly<br>I'm going to inhale press my Carriage<br>out let my chest drop down in between my<br>arms and then my exhale I'm going to<br>tuck my pelvis around through my spine<br>to come back in inhale press out let<br>your chest drop down X tail tuck around<br>to come in we have two more again after<br>this we're going to get a really good<br>workout in today last one<br>and then round and come in all right now<br>once we bring it back in we're going to<br>take our knees onto our carriages and<br>then our hands are going to go into our</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

</details>

Training Logs

Framework Versions

• Python: 3.13.2
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.7.0+cpu
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->