distil-cohere-l2-ckpt

SentenceTransformer based on google/bert_uncased_L-6_H-512_A-8

This is a sentence-transformers model finetuned from google/bert_uncased_L-6_H-512_A-8 on the parquet dataset. It maps sentences & paragraphs to a 512-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-6_H-512_A-8 <!-- at revision dd53ec6ca9d05e0a91b309c4e137f31988888071 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 512 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • parquet

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 512, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Just 320 FFs were constructed and production ceased in 1971.\nOther projects\nAustin A40 Sports: As one in a series of collaborations between Austin and Jensen, the Austin A40 Sports originated when Austin\'s chairman Leonard Lord, upon seeing the Interceptor, requested that Jensen, and their designer Eric Neale, develop a body that could use the A40 mechanicals.\nThe resulting body-on-frame A40 Sports\xa0– which debuted at the 1949 London Motor Show\xa0– had been designed by Eric Neale, an ex-Wolseley stylist who had joined Jensen in 1946. During production, the A40 Sports\' aluminium bodies were built by Jensen and transported to Austin\'s Longbridge plant for final assembly. The A40 Sports had been intended as more of a sporty touring car and not a sports car per se, and over 4000 examples were manufactured from 1951 to 1953.\nAustin-Healey 100: Although Jensen\'s design for a new Austin-based sports-car was rejected by the British Motor Corporation (BMC) in 1952 in favour of a design provided by Donald Healey, Jensen did win the BMC contract to build the bodies for the resultant Austin-Healey 100 and the rest of the "big Healey" cars. At the end of 1960 Austin-Healey cars occupied about 350 of the 850 men in Jensen\'s factory.',
    'Joshua Steele\'s disagreement, and subsequent correspondence, with Monboddo over details of the "melody and measure of speech" resulted in Steele\'s Prosodia Rationalis, a foundational work both in phonetics and in the analysis of verse rhythm.\nEvolutionary theorist\nMonboddo is considered by some scholars as a precursive thinker in the theory of evolution. However, some modern evolutionary historians do not give Monboddo an equally high standing in the influence of history of evolutionary thought.\n"Monboddo: Scottish jurist and pioneer anthropologist who explored the origins of language and society and anticipated principles of Darwinian evolution."\n"With some wavering, he extended Rousseau\'s doctrine of the identity of species of man and the chimp into the hypothesis of common descent of all the anthropoids, and suggested by implication a general law of evolution." Lovejoy.\nCharles Neaves, one of Monboddo\'s successors on the high court of Scotland, believed that proper credit was not given to Monboddo in evolutionary theory development. Neaves wrote in verse:\nThough Darwin now proclaims the law\nAnd spreads it far abroad, O!\nThe man that first the secret saw\nWas honest old Monboddo.\nThe architect precedence takes\nOf him that bears the hod, O!\nSo up and at them, Land of Cakes,',
    'As a consequence, in May 1833 Ozanam and a group of other young men founded the charitable Society of Saint Vincent de Paul, which already by the time of his death numbered upwards of 2,000 members. The founding members developed their method of service under the guidance of Sister Rosalie Rendu, a member of the Congregation of Daughters of Charity of Saint Vincent de Paul, who was prominent in serving the poor in the slums of Paris. The members of the conferences collaborated with Rendu during the time of the cholera epidemic. When fear had gripped the population, she organized the conferences in all the neighborhoods of Paris to care for the cholera victims, becoming well known in the city for her work, especially in the 12th arrondissement. Frederic\'s first act of charity was to take his supply of winter firewood and bring it to a widow whose husband had died of cholera.\nOzanam received the degrees of Bachelor of Laws in 1834, Bachelor of Arts in 1835 and Doctor of Laws in 1836. His father, who had wanted him to study law, died on 12 May 1837. Although he preferred literature, Ozanam worked in the legal profession in order to support his mother, and was admitted to the Bar in Lyon in 1837.\nIn 1835, Ozanam persuaded Monseigneur de Quélen, the Archbishop of Paris, to ask Jean-Baptiste Henri Lacordaire to preach a Lenten series at the Cathedral of Notre-Dame in Paris, as part of the Notre-Dame Lectures specially aimed at the catechesis of Christian youth, which had been inaugurated at the behest of his friend Ozanam. Lacordaire\'s first lecture took place on 8 March 1835, and was met with wide acclaim. Lacordaire was reputed to be the greatest pulpit orator of the nineteenth century. The social event of its day, it was well-attended and became an annual tradition in Paris. According to Thomas Bokenkotter, Lacordaire\'s Notre Dame Conferences, "...proved to be one of the most dramatic events of nineteenth century church history."',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6802, 0.6429],
#         [0.6802, 1.0000, 0.6874],
#         [0.6429, 0.6874, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

parquet

• Dataset: parquet
• Size: 3,810,976 training samples
• Columns: <code>text</code> and <code>label</code>
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	<ul><li>min: 99 tokens</li><li>mean: 314.97 tokens</li><li>max: 505 tokens</li></ul>	<ul><li>size: 1024 elements</li></ul>

• Samples:

  text	label
  <code>Negligent homicide is a criminal charge brought against a person who, through criminal negligence, allows another person to die. Other times, an intentional killing may be negotiated down to this lesser charge as a compromised resolution of a murder case, as might occur in the context of the intentional shooting of an unarmed man after a traffic altercation. Negligent homicide can be distinguished from involuntary manslaughter by its mens rea requirement: negligent homicide requires criminal negligence, while manslaughter requires recklessness.</code>	<code>[0.07659912109375, -0.01461029052734375, 0.008819580078125, -0.062469482421875, -0.0020580291748046875, ...]</code>
  <code>Latin trap is a subgenre of Latin hip hop music that originated in Puerto Rico. A direct descendant of southern hip hop, and influenced by reggaeton, it gained popularity after 2007, and has since spread throughout Latin America. The trap is slang for a place where drugs are sold. Latin trap is similar to mainstream trap with lyrics about life on la calle (the street), drugs, sex and violence.<br>Characteristics<br>Latin trap is a subgenre of Latin hip hop, taking influence from Southern hip hop as well as Puerto Rican genres like reggaeton and dembow. Vocals include a bend of rapping and singing using synthesizers and voice distorted autotune, often in Spanish, while still maintaining the trap style sonic circuitry. The lyrics in Latin trap are often about street life, violence, sex, drugs, and people who live on the other side of the law and are proud of it.<br>History</code>	<code>[-0.0191802978515625, -0.0106353759765625, -0.036407470703125, 0.029571533203125, -0.004840850830078125, ...]</code>
  <code>Surface 3 is a 2-in-1 detachable from the Microsoft Surface series, introduced by Microsoft in 2015. Unlike its predecessor, the Surface 2, Surface 3 utilizes an x86 Intel Atom system-on-chip architecture, or SoC, rather than a processor with ARM architecture such as the Nvidia Tegra that powered the Surface 2, and runs standard versions of Windows 8.1 or Windows 10.<br>History<br>The older, original Surface (also known as Surface RT) and Surface 2, with their ARM-based processors and Windows RT operating system, were designed to compete with the iPad and other tablets. The first Surface was criticized for performance issues, which were reduced in the succeeding Surface 2. Due to the differing processors, these devices were incompatible with the vast number of Windows programs written for x86-based computers, running only those written and compiled for Windows RT, loaded from Microsoft's application store.</code>	<code>[-0.04119873046875, -0.016265869140625, -0.0126800537109375, 0.0103912353515625, -0.0557861328125, ...]</code>

• Loss: <code>EmbedDistillLoss</code> with these parameters:

  {
      "distance_metric": "l2",
      "projection_dim": 1024
  }
  

Evaluation Dataset

parquet

• Dataset: parquet
• Size: 256 evaluation samples
• Columns: <code>text</code> and <code>label</code>
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	<ul><li>min: 53 tokens</li><li>mean: 308.2 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>size: 1024 elements</li></ul>

• Samples:

  text	label
  <code>Revolution Pro Wrestling (2022–present) Slater made his debut in Revolution Pro Wrestling at RevPro Live In London 64 on August 7, 2022, where he unsuccessfully challenged Luke Jacobs for the Undisputed British Cruiserweight Championship. He continued to make regular appearances in the company's signature events such as the British J-Cup, where he made his first appearance at the 202 edition where he defeated Lio Rush in the first rounds but fell short to Robbie X, Lee Hunter and Will Kaven in a four-way match in the finals which occurred on the same night. One year later, Slater succeeded in winning the cup by defeating Will Kaven in the first rounds ans then Harrison Bennett, Mascara Dorada and Wild Boar in the four-way match of the finals. At RevPro Revolution Rumble 2023 on March 26, Slater competed in the traditional royal rumble match, bout won by Michael Oku and also involving various other notable opponents such as Big Damo, Callum Newman, Eddie Dennis, Gabriel Kidd, Francesco ...</code>	<code>[-0.026824951171875, 0.03338623046875, 0.011566162109375, 0.01247406005859375, 0.002338409423828125, ...]</code>
  <code>On 9 April 2015, Masagos was promoted to the rank of full Minister but did not have a portfolio yet, so he was a Minister in the Prime Minister's Office. This was the first time in Singaporean history when there were two Malay ministers in the Cabinet, the other being Yaacob Ibrahim. Masagos was also promoted to Second Minister for Home Affairs and Second Minister for Foreign Affairs, and was put in charge of leading the PAP team in Tampines GRC. On the same day, Prime Minister Lee Hsien Loong said that Masagos's elevation to full Minister reflected the "progress of the Malay community" in Singapore. Masagos also said that he was honoured to have been appointed and that "[h]aving two Malay full ministers in the Cabinet for the first time in our nation's history reflects the [Government's] trust and recognition of the good progress made by the Malay-Muslim community".<br>On 1 October 2015, Masagos took up the portfolio of Minister for Environment and Water Resources. From 1 May 2018, he wa...</code>	<code>[0.039398193359375, -0.022613525390625, 0.006793975830078125, 0.00878143310546875, -0.005939483642578125, ...]</code>
  <code>In the late 15th century the Clan Mackay and Clan Ross had long been at feud. This resulted in the Battle of Tarbat in 1486 where the Mackays were defeated by the Rosses and chief Angus Roy Mackay, 9th of Strathnaver was killed. This was followed by the Battle of Aldy Charrish where the Rosses were defeated by the Mackays and the Ross chief was killed along with many of his clan. According to 17th-century historian Sir Robert Gordon, who was a younger son of Alexander Gordon, 12th Earl of Sutherland, the Clan Sutherland joined the side of the Clan Mackay at this battle. However, 19th-century historian Angus Mackay disputes the Sutherland's presence at the battle stating that it would be unlikely that the Earl of Sutherland at the time would have assisted against the Rosses as he was married to a daughter of the Ross chief of Balnagowan, and also that the feudal superiority of the Sutherlands over the Mackays "nowhere existed save in his own fertile imagination".<br>16th century and clan c...</code>	<code>[0.06927490234375, 0.021881103515625, 0.01392364501953125, -0.044342041015625, 0.058258056640625, ...]</code>

• Loss: <code>EmbedDistillLoss</code> with these parameters:

  {
      "distance_metric": "l2",
      "projection_dim": 1024
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• max_steps: 100000
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 2000
• fp16: True
• per_device_eval_batch_size: 32
• seed: 12

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 32
• num_train_epochs: 3.0
• max_steps: 100000
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 2000
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

<details><summary>Click to expand</summary> </details>

Training Time

• Training: 12.0 minutes
• Evaluation: 1.0 seconds
• Total: 12.0 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.7.0
• Transformers: 5.13.1
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

EmbedDistillLoss

@article{kim2023embeddistill,
    title={EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval},
    author={Kim, Seungyeon and Rawat, Ankit Singh and Zaheer, Manzil and Jayasumana, Sadeep and Sadhanala, Veeranjaneyulu and Jitkrittum, Wittawat and Menon, Aditya Krishna and Fergus, Rob and Kumar, Sanjiv},
    year={2023},
    eprint={2301.12005},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->