timely-arctic-large

Technical Report and Model Pipeline

To access our technical report and model pipeline scripts visit our github

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l <!-- at revision b7c623b8902f02627a9420b73b2fd6300aad7a68 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Represent this sentence for searching relevant passages: Nov 6 2002 Easter seals (philately)',
    '06/11/2002 An Easter seal is a form of charity label issued to raise funds for charitable purposes. They are issued by the Easterseals charity in the United States, and by the Canadian Easter Seals charities. Easter seals are applied to the front of mail to show support for particular charitable causes. They are distributed along with appeals to donate to the charities they support. Easter seals are a form of Cinderella stamp. They do not have any postal value. Cinderella stamps\n',
    '2017 Winter The Waterfall Model was the first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model. ... The Waterfall model is the earliest SDLC approach that was used for software development. The waterfall Model illustrates the software development process in a linear sequential flow.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 55,736 training samples
• Columns: <code>anchors</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchors	positive
  type	string	string
  details	<ul><li>min: 14 tokens</li><li>mean: 20.25 tokens</li><li>max: 33 tokens</li></ul>	<ul><li>min: 15 tokens</li><li>mean: 47.2 tokens</li><li>max: 75 tokens</li></ul>

• Samples:

  anchors	positive
  <code>Represent this sentence for searching relevant passages: are bugs attracted to citronella November 10?</code>	<code>Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10<br></code>
  <code>Represent this sentence for searching relevant passages: are bugs attracted to citronella 11/10/09?</code>	<code>Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10/09<br></code>
  <code>Represent this sentence for searching relevant passages: are bugs attracted to citronella Jan 15?</code>	<code>Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 01/15<br></code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,000 evaluation samples
• Columns: <code>anchors</code> and <code>positive</code>
• Approximate statistics based on the first 1000 samples:

  	anchors	positive
  type	string	string
  details	<ul><li>min: 12 tokens</li><li>mean: 21.64 tokens</li><li>max: 40 tokens</li></ul>	<ul><li>min: 20 tokens</li><li>mean: 66.86 tokens</li><li>max: 512 tokens</li></ul>

• Samples:

  anchors	positive
  <code>Represent this sentence for searching relevant passages: today:2068-02-10 what is the meaning of the idiom put two and two together last monday </code>	<code>put two and two together. to understand something by using the information you have: I didn't tell her George had left, but she noticed his car was gone and put two and two together. (Definition of put two and two together from the Cambridge Academic Content Dictionary © Cambridge University Press) 02/06/2068<br></code>
  <code>Represent this sentence for searching relevant passages: Complete the Throat of the World Quest in Skyrim</code>	<code>The Throat of the World is the fifth quest in the second act of the Skyrim’s main quest. During this mission, all the mystery about the game’s main antagonist, Alduin, will be revealed to you.<br></code>
  <code>Represent this sentence for searching relevant passages: are blanco kitchen faucets good 04/13/86?</code>	<code>Nevertheless, these are good to very good faucets built with good quality components throughout, backed by a strong warranty and superior customer service from a well-established company. Blanco sells only kitchen, prep and bar faucets, nothing for the bathroom. Apr 13 1986<br></code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 1e-06
• weight_decay: 0.01
• num_train_epochs: 1
• warmup_ratio: 0.1
• warmup_steps: 400
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-06
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 400
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.43.4
• PyTorch: 2.4.0+cu121
• Accelerate: 0.33.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->