XLMRoBERTaM3-CustomPoolin-v1.02-1024dMLP-s1

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the NLI, natural-questions, vitaminc, xsum, paws and global_dataset datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3 <!-- at revision 5617a9f61b028005a4858fdac845db406aefb181 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Datasets:
  • NLI
  • natural-questions
  • vitaminc
  • xsum
  • paws
  • global_dataset
• Language: en

<!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): AdvancedWeightedPooling(
    (mha): MultiheadAttention(
      (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
    )
    (MLP): Sequential(
      (0): SwiGLUBlock(
        (in_proj_swish): Linear(in_features=1024, out_features=2048, bias=True)
        (in_proj_gate): Linear(in_features=1024, out_features=2048, bias=True)
      )
      (1): Dropout(p=0.05, inplace=False)
      (2): Linear(in_features=2048, out_features=1024, bias=True)
    )
    (layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bobox/XLMRoBERTaM3-CustomPoolin-v1.02-1024dMLP-s1")
# Run inference
sentences = [
    'In mathematical astronomy , his fame is due to the introduction of the astronomical globe , and his early contributions to understanding the movement of the planets .',
    'His fame is due in mathematical astronomy to the introduction of the astronomical globe and to his early contributions to the understanding of the movement of the planets .',
    'In 1994 , Rodrigo Leão left the band to start a solo career , replaced by Carlos Maria Trindade ( keyboard synthesizer ) .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-test
• Evaluated with <code>EmbeddingSimilarityEvaluator</code>

Binary Classification

• Datasets: allNLI-dev and Qnli-dev
• Evaluated with <code>BinaryClassificationEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Datasets

<details><summary>NLI</summary>

NLI

• Dataset: NLI at d43e6fe
• Size: 750 training samples
• Columns: <code>anchor</code>, <code>entailment</code>, and <code>negative</code>
• Approximate statistics based on the first 750 samples:

  	anchor	entailment	negative
  type	string	string	string
  details	<ul><li>min: 5 tokens</li><li>mean: 24.9 tokens</li><li>max: 176 tokens</li></ul>	<ul><li>min: 5 tokens</li><li>mean: 16.4 tokens</li><li>max: 54 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 16.53 tokens</li><li>max: 49 tokens</li></ul>

• Samples:

  anchor	entailment	negative
  <code>09:00 On Thursday, AT&T said they have teamed with Juniper Networks to develop a mobile security platform for both businesses and consumers.</code>	<code>AT&T and Juniper to develop mobile security platform</code>	<code>AT&T and Juniper disassemble mobile security platform</code>
  <code>two police motorcycles driving down the road in front of a cop car</code>	<code>Two motorcycle cops and a police car on a street. </code>	<code>No motorcycle cops and a police car on a street.</code>
  <code>I've told you about their size.</code>	<code>I have told you about their size.</code>	<code>I have not told you about their size.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>natural-questions</summary>

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 750 training samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 750 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 10 tokens</li><li>mean: 13.32 tokens</li><li>max: 25 tokens</li></ul>	<ul><li>min: 17 tokens</li><li>mean: 148.26 tokens</li><li>max: 651 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>who is winner in bigg boss season 5 kannada</code>	<code>Bigg Boss Kannada 5 Bigg Boss Kannada 5 (BBK5) was the fifth season of the Kannada television series Bigg Boss Kannada, that premiered on 15 October 2017.[1] Sudeep reprised his role as the host of the show.[2] The finale of the season took place 28 January 2018, and rapper Chandan Shetty was declared the winner of the show and the prize money of ₹50 lakh. Sales representative Diwaker was voted the runner-up.[3]</code>
  <code>what side of the street do they drive on in sweden</code>	<code>Left- and right-hand traffic Sweden was LHT from about 1734 to 1967,[17] despite having land borders with RHT countries, and approximately 90 percent of cars being left-hand drive (LHD) vehicles.[18] A referendum was held in 1955, with an overwhelming majority voting against a change to RHT. Nevertheless, some years later the government ordered a conversion, which took place at 5 am on Sunday, 3 September 1967. The accident rate dropped sharply after the change,[19] but soon rose back to near its original level.[20] The day was known as Dagen H ("H-Day"), the 'H' being for Högertrafik or right traffic. When Iceland switched the following year, it was known as H-dagurinn, again meaning "H-Day".[21]</code>
  <code>what is the difference between mandelbrot and biscotti</code>	<code>Mandelbrot (cookie) Its precise origin is unknown, as is its historic relationship with biscotti, a similar Italian cookie. While mandelbrot and biscotti both have a crunchy exterior, mandelbrot is slightly softer than biscotti due to its higher oil and/or butter content.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>vitaminc</summary>

vitaminc

• Dataset: vitaminc at be6febb
• Size: 370,653 training samples
• Columns: <code>claim</code> and <code>evidence</code>
• Approximate statistics based on the first 1000 samples:

  	claim	evidence
  type	string	string
  details	<ul><li>min: 9 tokens</li><li>mean: 20.44 tokens</li><li>max: 73 tokens</li></ul>	<ul><li>min: 9 tokens</li><li>mean: 44.61 tokens</li><li>max: 191 tokens</li></ul>

• Samples:

  claim	evidence
  <code>The Script is a pop band .</code>	<code>The Script are an Irish pop band formed in 2007 in Dublin , Ireland .</code>
  <code>Scott Skiles scored fewer than 55 points in home games .</code>	<code>He set several records during high school , including most points in a home game ( 53 ) and most points in an away game ( 56 ) .</code>
  <code>The Black Cauldron was released before July 25 , 1985 .</code>	<code>The film was distributed theatrically through Buena Vista Distribution on July 24 , 1985 .</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>xsum</summary>

xsum

• Dataset: xsum at 044020f
• Size: 131,779 training samples
• Columns: <code>summary</code> and <code>document</code>
• Approximate statistics based on the first 1000 samples:

  	summary	document
  type	string	string
  details	<ul><li>min: 11 tokens</li><li>mean: 30.72 tokens</li><li>max: 62 tokens</li></ul>	<ul><li>min: 63 tokens</li><li>mean: 311.14 tokens</li><li>max: 550 tokens</li></ul>

• Samples:

  summary	document
  <code>The amount of time spent needing daily care in late life has doubled in England over the past two decades, a study suggests.</code>	<code>The Newcastle University study found men spent 2.4 years on average needing regular care and women three years.<br>This includes everything from help with washing and dressing each day to round-the-clock care.<br>Researchers said it suggested there needed to be a sharp increase in the number of care home places to cope.<br>It comes as ministers consider a new way to fund the system.<br>The government has promised major reform amid reports that councils are struggling to provide enough support to cope with the ageing population.<br>The latest research, published in the Lancet, looked at not just the growth in the numbers of older people but also how many of those years were spent needing daily care.<br>Between 1991 and 2011, life expectancy increased by more than four years for both men and women to 82.6 and 85.6 respectively.<br>But the number of those years spent with substantial care needs rose much more rapidly, from 1.1 to 2.4 for men and 1.6 to three for women.<br>Looking ahead to 2025, it means there wi...</code>
  <code>A man has admitted sexually assaulting two women in the same street two months apart.</code>	<code>Craig Perkins had initially denied being involved in the attacks in Bournemouth's Boundary Road in September and December of last year.<br>But on Wednesday at Bournemouth Crown Court he pleaded guilty to two counts of sexual assault.<br>The 29-year-old, of Victoria Park Road, Bournemouth, has been remanded in custody and will be sentenced on 5 May.<br>Police said the victims were both in their 20s - the first was assaulted on Tuesday 13 September and the second attack happened on Thursday 24 November.<br>Perkins was arrested on 14 December.<br>Det Ch Insp Sarah Derbyshire, of Dorset Police's major crime investigation team, said: "Stranger sex attacks such as these are very rare in Dorset and we are committed toward investigating them thoroughly and bringing the offender to justice.<br>"The victims in this case have been updated about Perkins' guilty pleas and I would like to pay tribute to them for having the confidence to report these offences to Dorset Police and the assistance they have given to the ...</code>
  <code>Durham produced a below-par batting display as they lost by seven wickets to Worcestershire in the One-Day Cup.</code>	<code>A 22 overs-a-side game was all that was possible after a long rain delay, but the home side were bowled out for 90.<br>Mark Stoneman top-scored with 29 and the only other batsman to reach double figures was Paul Collingwood (17).<br>Chris Rushworth took 3-19 as the visitors began their reply, but Alexei Kervezee (37) and Brett D'Oliveira (20) saw them to 91-3 with 17 balls in hand.<br>Their unbroken partnership was worth 60 after Kervezee collected the winning single from the bowling of Usman Arshad.<br>Durham reached 35-1 at the start of their innings after play got under way at 15:30 BST, but then lost four wickets for 11 runs.<br>D'Oliveira, Ed Barnard, Joe Leach and Chris Russell took two wickets each as they were finally dismissed at the start of the 22nd over.<br>Durham's total was their seventh-lowest in non-Twenty20 limited-overs matches games.<br>Rushworth exploited the conditions superbly at the start of Worcestershire's innings, but once he was out of the attack, Kervezee and D'Oliveira were abl...</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>paws</summary>

paws

• Dataset: paws at 161ece9
• Size: 49,401 training samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 12 tokens</li><li>mean: 30.94 tokens</li><li>max: 56 tokens</li></ul>	<ul><li>min: 11 tokens</li><li>mean: 30.97 tokens</li><li>max: 55 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>Charley Frazier ( born August 12 , 1939 in Houston , Texas ) is a former American Football Wide Receiver from the NFL and the American Football League .</code>	<code>Charley Frazier ( born August 12 , 1939 in Houston , Texas ) is a former American football receiver in the American Football League and the NFL .</code>
  <code>Indonesian dumplings were influenced and brought by Chinese immigrants to Indonesia .</code>	<code>Indonesian dumplings were influenced and brought to Indonesia by Chinese immigrants .</code>
  <code>The SSSI has an area of 190.3 hectares , while the SAC has 168.3 hectares .</code>	<code>The SSSI has an area of 190.3 hectares while the SAC covers 168.3 hectares .</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>global_dataset</summary>

global_dataset

• Dataset: global_dataset
• Size: 71,250 training samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 7 tokens</li><li>mean: 24.45 tokens</li><li>max: 115 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 106.19 tokens</li><li>max: 564 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>Taobao is a Chinese online shopping site similar to eBay , Amazon and Rakuten , which is operated by Alibaba Group in Hangzhou , Zhejiang .</code>	<code>Taobao is a Chinese online shopping website similar to eBay , Amazon and Rakuten , which is operated in Hangzhou , Zhejiang by Alibaba Group .</code>
  <code>Because of the lack of wood , boats were bundled with made papyrus reeds .</code>	<code>Because of the lack of wood , boats with papyrus reeds were bundled .</code>
  <code>New Zealand leg-spinner Ish Sodhi hopes his stint playing in Nottinghamshire's T20 campaign this summer will lead to a longer stay in England.</code>	<code>The 24-year-old has played 41 international matches in all formats.<br>He has been particularly effective in T20, with 21 wickets at 14.47 and a strike-rate of a wicket every 13 balls.<br>"In the last year or so I have definitely been a lot more successful in the T20 stuff than in the other stuff," he told BBC Radio Nottingham.<br>"But in the last six months I have been finding my way in the four-stuff and one-dayers.<br>"In the future I would love to come over and play all the forms. At this stage the T20 is the main focus. It is still a wee, wee way away but I will continue to look to hone my T20 skills and try to be in tip-top condition when I come over."<br>Sodhi says playing in England is a "great opportunity" with the 2019 World Cup in mind.<br>"It's great to play at these grounds where I will potentially play a World Cup, which I am targeting," he said.<br>"It will be great to get used to conditions. The opportunity came up and I will try to grab it with both hands."<br>Sodhi will be the second of two ...</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details>

Evaluation Datasets

<details><summary>NLI</summary>

NLI

• Dataset: NLI at d43e6fe
• Size: 85 evaluation samples
• Columns: <code>anchor</code>, <code>entailment</code>, and <code>negative</code>
• Approximate statistics based on the first 85 samples:

  	anchor	entailment	negative
  type	string	string	string
  details	<ul><li>min: 9 tokens</li><li>mean: 17.02 tokens</li><li>max: 36 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 12.96 tokens</li><li>max: 25 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 13.53 tokens</li><li>max: 26 tokens</li></ul>

• Samples:

  anchor	entailment	negative
  <code>The girls walk down the street.</code>	<code>Girls walk down the street.</code>	<code>Girls do not walk down the street.</code>
  <code>Two computers sitting on top of a desk.</code>	<code>A laptop computer and a desktop computer on a white desk</code>	<code>A laptop computer and a desktop computer on a black desk</code>
  <code>A bathroom with a toilette with it's seat down.</code>	<code>A bathroom with a sink and a toilet</code>	<code>A bathroom without a sink or a toilet</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>natural-questions</summary>

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 113 evaluation samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 113 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 10 tokens</li><li>mean: 13.57 tokens</li><li>max: 23 tokens</li></ul>	<ul><li>min: 34 tokens</li><li>mean: 176.3 tokens</li><li>max: 2497 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>kiss him not me where does the anime end in the manga</code>	<code>Kiss Him, Not Me Kiss Him, Not Me, known in Japan as Watashi ga Motete Dōsunda (Japanese: 私がモテてどうすんだ, Hepburn: lit. What's the Point of Me Getting Popular?), is a Japanese romantic comedy shōjo manga series written and illustrated by Junko.[2] It is published by Kodansha since 2013 on Bessatsu Friend magazine.[3] Twelve volumes compiling the chapters have been released so far.[2] It is published online in English by Crunchyroll and the volumes will be published by Kodansha USA.[3] An audio drama adaptation of the first chapter was released on January 13, 2015.[4] An anime adaptation by Brain's Base aired in Japan between October and December 2016.[5][6] The manga won Best Shōjo Manga at the 40th Kodansha Manga Awards.</code>
  <code>who sings i just want to use your love</code>	<code>Your Love (The Outfield song) "Your Love" is a song by the English rock band the Outfield, taken from their debut album Play Deep (1985). The song was penned by the band's guitarist John Spinks.</code>
  <code>how many episodes of westworld are in season 1</code>	<code>Westworld (season 1) The first season of the American science fiction western television series Westworld (subtitled The Maze) premiered on HBO on October 2, 2016, and concluded on December 4, 2016. It consisted of ten episodes, each running approximately 60 minutes in length and was broadcast on Sundays in the United States. The complete first season was released on home media on November 7, 2017.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>vitaminc</summary>

vitaminc

• Dataset: vitaminc at be6febb
• Size: 63,054 evaluation samples
• Columns: <code>claim</code> and <code>evidence</code>
• Approximate statistics based on the first 1000 samples:

  	claim	evidence
  type	string	string
  details	<ul><li>min: 10 tokens</li><li>mean: 22.35 tokens</li><li>max: 48 tokens</li></ul>	<ul><li>min: 11 tokens</li><li>mean: 37.94 tokens</li><li>max: 75 tokens</li></ul>

• Samples:

  claim	evidence
  <code>More than 273 people have died from the 2019-20 coronavirus outside mainland China .</code>	<code>More than 3,200 people have died : almost 3,000 in mainland China and around 275 in other countries .</code>
  <code>More than 146,500 people have been infected with coronavirus globally , during the 2019�20 pandemic .</code>	<code>more than 147,000 cases have been confirmed worldwide .</code>
  <code>Over 278,000 coronavirus cases had been confirmed around the world by March 21 , 2020 .</code>	<code>As of 21 March , more than 278,000 cases of COVID-19 have been reported in over 186 countries and territories , resulting in more than 11,500 deaths and 92,000 recoveries. virus seems to mostly spread between people via respiratory droplets .</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>xsum</summary>

xsum

• Dataset: xsum at 044020f
• Size: 131,779 evaluation samples
• Columns: <code>summary</code> and <code>document</code>
• Approximate statistics based on the first 1000 samples:

  	summary	document
  type	string	string
  details	<ul><li>min: 17 tokens</li><li>mean: 30.71 tokens</li><li>max: 43 tokens</li></ul>	<ul><li>min: 70 tokens</li><li>mean: 305.77 tokens</li><li>max: 543 tokens</li></ul>

• Samples:

  summary	document
  <code>A new species of moss has been found growing on 10 maple trees in a Carmarthenshire car park, but experts are in two minds about its origins.</code>	<code>Welsh Bristle-moss was discovered near Dryslwyn Castle, close to Llandeilo, by the Countryside Council for Wales.<br>It said it might have evolved from a genetically similar moss.<br>But it could be an undiscovered species that was imported from the Continent on maples used to landscape the car park in the 1990s.<br>There are about 900 species of moss in Britain and 587 of those are found in Wales.<br>The Welsh Bristle-moss was discovered during a survey which is recording mosses growing on trees in south Wales.<br>Experts said the moss had a unique combination of distinctive traits. It differed from related mosses because of its round-tipped leaf tips and flat leaf edges.<br>Countryside Council for Wales (CCW) moss ecologist Sam Bosanquet, who made the new find, said: "Welsh Bristle-moss highlights the need to be ever vigilant and open-minded, even when looking at plants in mundane places like car parks.<br>"This is a high-point in our regular work of recording mosses which grow on trees in south Wales.<br>"...</code>
  <code>A former Soviet army officer has been convicted by a US jury of planning and leading a Taliban attack on American forces in Afghanistan in 2009.</code>	<code>The jury found Irek Hamidullin guilty on 15 counts, including supporting terrorists and conspiracy to use a weapon of mass destruction.<br>The 55-year-old is the first military prisoner from Afghanistan to be tried in a US federal court.<br>Some of the charges carry a mandatory life sentence.<br>About 30 insurgents died in the attack, with Hamidullin the only survivor, while no American or Afghan soldiers were killed.<br>Hamidullin, who did not testify during the trial, is expected to be sentenced on 6 November.<br>Lawyers say it is unusual for someone captured on the battlefield in Afghanistan to be transferred to the United States for trial in a federal court.<br>Hamidullin's defence lawyers had tried unsuccessfully to have the charges dismissed, saying their client was a prisoner of war and ineligible for trial in civilian court.<br>Prosecutors argued federal law protected US soldiers no matter where they were.<br>The jury in Richmond. Virginia, reached its verdict after five days of testimony and eight ho...</code>
  <code>UK troops could be deployed to train moderate Syrian rebels in the fight against Islamic State militants (IS), the defence secretary has said.</code>	<code>Michael Fallon told BBC News that UK troops could be sent to a country neighbouring Syria, possibly Jordan.<br>He insisted however that UK forces would not engage in direct combat.<br>The US is leading efforts to train a Syrian opposition to fight IS, also known as ISIS, which has captured large parts of of the country.<br>The country's National Security Adviser Susan Rice said a deal had been reached with Turkey to allow the US to train Syrian rebels on its soil, although this has been denied by Turkish officials.<br>Mr Fallon discussed the possibility of launching training operations, while visiting the Royal Fleet Auxiliary Ship, Argus, in Falmouth.<br>A specialist team of 12 soldiers from the Yorkshire Regiment is already training Kurdish fighters in Iraq to use UK-supplied heavy machine guns.<br>And the UK is to fund bomb disposal training for the Kurdish Peshmerga forces to counter the threat of Improvised Explosive Devices (IEDs), Foreign Secretary Philip Hammond announced on Monday.<br>The Prime Mi...</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>paws</summary>

paws

• Dataset: paws at 161ece9
• Size: 8,000 evaluation samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 11 tokens</li><li>mean: 31.67 tokens</li><li>max: 56 tokens</li></ul>	<ul><li>min: 10 tokens</li><li>mean: 31.33 tokens</li><li>max: 54 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>He also wrote a large number of vocal arrangements and orchestral accompaniments to varieties .</code>	<code>He also wrote a large number of vocal arrangements and orchestral accompaniments for varieties .</code>
  <code>In 1994 , Rodrigo Leão left the band to start a solo career , being replaced by Carlos Maria Trindade ( keyboard synthesizer ) .</code>	<code>In 1994 , Rodrigo Leão left the band to start a solo career , replaced by Carlos Maria Trindade ( keyboard synthesizer ) .</code>
  <code>Until 1951 , he was active as a socialist in post-war legislation when he decided to focus on local politics .</code>	<code>He was active as a socialist in the post-war legislature until 1951 , when he decided to focus on local politics .</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details> <details><summary>global_dataset</summary>

global_dataset

• Dataset: global_dataset
• Size: 256 evaluation samples
• Columns: <code>sentence1</code> and <code>sentence2</code>
• Approximate statistics based on the first 256 samples:

  	sentence1	sentence2
  type	string	string
  details	<ul><li>min: 10 tokens</li><li>mean: 23.68 tokens</li><li>max: 51 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 112.38 tokens</li><li>max: 511 tokens</li></ul>

• Samples:

  sentence1	sentence2
  <code>All babies born from Tuesday across the UK will have an anti-hepatitis B injection added to the other routine vaccinations they are given in their early life.</code>	<code>The jab protects against viral infections that cause cirrhosis and liver cancer.<br>Babies are already vaccinated against diphtheria, tetanus, whooping cough, Hib and polio.<br>Public Health England said the new vaccine had been "shown to be safe".<br>Babies are currently given vaccinations when they are eight, 12 and 16 weeks old and the new injection will be given at the same time as the others.<br>Previously, the hepatitis B vaccine was available on the NHS as a separate jab and was only administered to infants considered at risk, such as those born to infected mothers.<br>While hepatitis B rates in the UK are generally very low, in some inner city areas up to 1% of antenatal women are infected.<br>The infection has no symptoms so many of these women will be unaware they are ill, while their babies are considered at high risk.<br>Mary Ramsay, head of immunisation at Public Health England, said: "The Hexavalent vaccine has been extensively tested and shown to be safe and is widely used internationally wi...</code>
  <code>A black man in a long sleeves white collared shirt and a tie is walking to work in a big city.</code>	<code>The man is wearing work attire and is walking to his job.</code>
  <code>ACVM is based in Glasgow and has offices in Edinburgh , Aberdeen , Newcastle , Manchester and Milton Keynes .</code>	<code>ACVM is based in Glasgow and has subsidiaries in Edinburgh , Aberdeen , Newcastle , Manchester and Milton Keynes .</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.02}
  

</details>

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 192
• per_device_eval_batch_size: 256
• learning_rate: 0.0001
• weight_decay: 0.001
• lr_scheduler_type: cosine_with_min_lr
• lr_scheduler_kwargs: {'num_cycles': 0.5, 'min_lr': 3.3333333333333335e-05}
• warmup_ratio: 0.15
• save_safetensors: False
• fp16: True
• remove_unused_columns: False
• push_to_hub: True
• hub_model_id: bobox/XLMRoBERTaM3-CustomPoolin-v1.02-1024dMLP-s1-checkpoints-tmp
• hub_strategy: all_checkpoints
• hub_private_repo: False
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 192
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.001
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: cosine_with_min_lr
• lr_scheduler_kwargs: {'num_cycles': 0.5, 'min_lr': 3.3333333333333335e-05}
• warmup_ratio: 0.15
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: False
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: bobox/XLMRoBERTaM3-CustomPoolin-v1.02-1024dMLP-s1-checkpoints-tmp
• hub_strategy: all_checkpoints
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>

The model card is truncated. Read the rest on Hugging Face.