PurpleStatic

PurpleStatic: Static Embeddings

This is a sentence-transformers model trained on the BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1, Heliosoph/Quora-Question-Pairs, owenkaplinsky/wildchat-paraphrases, mjbommar/ogbert-v1-contrastive and mjbommar/opengloss-v1.3-contrastive-examples datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: inf tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1
  • Heliosoph/Quora-Question-Pairs
  • owenkaplinsky/wildchat-paraphrases
  • mjbommar/ogbert-v1-contrastive
  • mjbommar/opengloss-v1.3-contrastive-examples
• Language: en
• License: wtfpl

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): StaticEmbedding({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("LocalWisdom/PurpleStatic")
# Run inference
sentences = [
    'How do you write a song?',
    'How do I write a song?',
    'How did Portugal become an independent country from Spain?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8566, 0.1409],
#         [0.8566, 1.0000, 0.1259],
#         [0.1409, 0.1259, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Datasets

<details><summary>BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1</summary>

BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

• Dataset: BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1
• Size: 5,694 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 47 characters</li><li>mean: 135.34 characters</li><li>max: 352 characters</li></ul>	<ul><li>min: 70 characters</li><li>mean: 146.17 characters</li><li>max: 322 characters</li></ul>

• Samples:

  anchor	positive
  <code>Of our total net revenue of $10.8 billion in the fiscal year ended October 31, 2000, we generated 44.2% in the United States and 55.8% internationally.</code>	<code>In the fiscal year concluding October 31, 2000, the company produced 44.2% of its $10.8 billion total net revenue within the United States and 55.8% from international markets.</code>
  <code>Our sales strategy is to sell to and service our largest accounts (hospital and corporate business) directly while employing third-party distributors and manufacturer's representatives for smaller or more geographically dispersed countries.</code>	<code>We utilize a sales approach where we directly manage and sell to our major hospital and corporate clients, while relying on external distributors and manufacturer agents to reach smaller markets or those in remote locations.</code>
  <code>In August 2000, we announced a restructuring of our healthcare solutions business.</code>	<code>The company declared a reorganization of its healthcare solutions division in August 2000.</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

</details> <details><summary>Heliosoph/Quora-Question-Pairs</summary>

Heliosoph/Quora-Question-Pairs

• Dataset: Heliosoph/Quora-Question-Pairs
• Size: 149,263 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 16 characters</li><li>mean: 54.75 characters</li><li>max: 139 characters</li></ul>	<ul><li>min: 21 characters</li><li>mean: 54.34 characters</li><li>max: 127 characters</li></ul>

• Samples:

  anchor	positive
  <code>Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?</code>	<code>I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?</code>
  <code>How can I be a good geologist?</code>	<code>What should I do to be a great geologist?</code>
  <code>How do I read and find my YouTube comments?</code>	<code>How can I see all my Youtube comments?</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

</details> <details><summary>owenkaplinsky/wildchat-paraphrases</summary>

owenkaplinsky/wildchat-paraphrases

• Dataset: owenkaplinsky/wildchat-paraphrases
• Size: 1,383,750 training samples
• Columns: <code>anchor</code> and <code>positive</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	<ul><li>min: 20 characters</li><li>mean: 133.6 characters</li><li>max: 733 characters</li></ul>	<ul><li>min: 23 characters</li><li>mean: 129.4 characters</li><li>max: 639 characters</li></ul>

• Samples:

  anchor	positive
  <code>name current finance minister of pakistan</code>	<code>identify the present finance minister of pakistan</code>
  <code>name current finance minister of pakistan</code>	<code>who is pakistan's finance minister right now</code>
  <code>name current finance minister of pakistan</code>	<code>state the name of the current pakistani minister of finance</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

</details> <details><summary>mjbommar/ogbert-v1-contrastive</summary>

mjbommar/ogbert-v1-contrastive

• Dataset: mjbommar/ogbert-v1-contrastive
• Size: 2,673,774 training samples
• Columns: <code>anchor</code> and <code>positives</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positives
  type	string	string
  modality	text	text
  details	<ul><li>min: 4 characters</li><li>mean: 50.25 characters</li><li>max: 178 characters</li></ul>	<ul><li>min: 4 characters</li><li>mean: 36.68 characters</li><li>max: 121 characters</li></ul>

• Samples:

  anchor	positives
  <code>The arrangement of halftone dots or grid that encodes tonal values in a halftone image.</code>	<code>The halftone dot grid encodes tonal levels.</code>
  <code>A formal subset of managers designated to handle day-to-day operations and implement board decisions.</code>	<code>The executive team meets monthly to review progress and adjust plans.</code>
  <code>niche market segment</code>	<code>submarket</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

</details> <details><summary>mjbommar/opengloss-v1.3-contrastive-examples</summary>

mjbommar/opengloss-v1.3-contrastive-examples

• Dataset: mjbommar/opengloss-v1.3-contrastive-examples
• Size: 161,496 training samples
• Columns: <code>anchor</code>, <code>positives</code>, <code>negatives_1</code>, and <code>negatives_2</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positives	negatives_1	negatives_2
  type	string	string	string	string
  modality	text	text	text	text
  details	<ul><li>min: 37 characters</li><li>mean: 70.41 characters</li><li>max: 145 characters</li></ul>	<ul><li>min: 36 characters</li><li>mean: 71.77 characters</li><li>max: 146 characters</li></ul>	<ul><li>min: 33 characters</li><li>mean: 72.34 characters</li><li>max: 145 characters</li></ul>	<ul><li>min: 35 characters</li><li>mean: 71.88 characters</li><li>max: 146 characters</li></ul>

• Samples:

  anchor	positives	negatives_1	negatives_2
  <code>Field interviews revealed a stony demeanor even when pressed for clarification.</code>	<code>Field interviews revealed a stoic demeanor even when pressed for clarification.</code>	<code>Field interviews revealed a warm demeanor even when pressed for clarification.</code>	<code>Field interviews revealed an empathetic demeanor even when pressed for clarification.</code>
  <code>The board followed formal protocol during the review.</code>	<code>The board followed official protocol during the review.</code>	<code>The board followed casual protocol during the review.</code>	<code>The board followed informal protocol during the review.</code>
  <code>The athlete carefully inspected the luge before the final run.</code>	<code>The athlete carefully inspected the toboggan before the final run.</code>	<code>The athlete carefully inspected the sled before the final run.</code>	<code>The athlete carefully inspected the bobsled before the final run.</code>

• Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

</details>

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4096
• num_train_epochs: 20
• learning_rate: 0.0001
• warmup_steps: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 4096
• num_train_epochs: 20
• max_steps: -1
• learning_rate: 0.0001
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Training Time

• Training: 2.1 hours

Framework Versions

• Python: 3.13.11
• Sentence Transformers: 5.6.1
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->