finetuned-modernbert-finance-large

SentenceTransformer based on tyuan73/ModernBERT-large-with-new-tokenizer

This is a sentence-transformers model finetuned from tyuan73/ModernBERT-large-with-new-tokenizer on the processed_yahoo_finance_stockmarket_news dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: tyuan73/ModernBERT-large-with-new-tokenizer <!-- at revision bf522aad8a5b444e01ae740df8725f3a6083ba4c -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • processed_yahoo_finance_stockmarket_news

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tyuan73/finetuned-modernbert-finance-large")
# Run inference
sentences = [
    "Stocks will rally 5% in just 5 days after the Fed's rate decision this week, Fundstrat says",
    'Photo by Cindy Ord/Getty Images for YahooThe stock market could see a 5% gain over the next week, according to Fundstrat\'s Tom Lee.The rally would be sparked by a dovish Fed FOMC meeting on Wednesday that all but confirms imminent interest rate cuts."These are significant gains, implying the S&P 500 could gain 200-300 points in the next week," Lee said.The stock market is poised to surge as much as 5% in the next week, according to a Wednesday note from Fundstrat.The research firm said it expects an explosive rally in theS&P 500to materialize in the five days following the Federal Reserve\'s FOMC meeting on Wednesday.While the Fed isnot expected to cut interest rates at its July FOMC meeting,it is expected to signal that a rate cut is all but certain when it meets again in September."The key premise is the Fed is likely to commit to a September rate cut of at least 25bp. A possibility of more than that is not necessary. And while bond markets have priced in 100% probability of this, equity investors likely will not be convinced until the Fed affirms this as such," Fundstrat\'s Tom Lee said.Thenear certainty of a rate cut from the Fedin September should spark a risk-on rally for stocks, especially given that theNasdaq 100has already experienced a near-10% correctionin recent weeks, according to the note."Overall, we believe risk-on moment is coming," Lee said.Lee\'s confidence in a strong rally post-Fed meeting is based on the fact that recent Fed meetings have sparked a big rally in stocks.In the past two years, when stocks were down heading into a Fed FOMC meeting, stocks saw a five-day gain of as much as 5.5% and a median gain of 3.4%."These are significant gains, implying the S&P 500 could gain 200-300 points in the next week. This is very compelling in our view," Lee said.And while a 25 basis point interest rate cut may not seem like much, it has real-world economic impacts that could ultimately influence the US housing market in a big way."Here are some tangible reasons a Fed cut makes sense: 30-year mortgage has excess spread to 10-year due to uncertainty. The spread could shrink from 270 basis points to 170 basis points (50-year average)," Lee explained.Interest rate cuts from the Fed, however small, would also help alleviate an ongoing slowdown in the housing, durables, and auto markets, Lee said.A 5% rally in the S&P 500 would catapult the index to fresh record highs, completely erasing its 5% decline over the past few weeks.Read the original article onBusiness Insider',
    'AMBALA, India (Reuters) - Indian farmers demanding higher prices for their crops will postpone a planned protest march to New Delhi until unions hold another round of talks with government ministers on Sunday.\n\nAgriculture Minister Arjun Munda, who met farmers\' representatives on Thurdsay along with Commerce Minister Piyush Goyal and Minister of State for Home Affairs Nityanand Rai, said the talks were "positive".\n\n"We have decided that the next meeting to take the discussion forward will take place on Sunday at 6 pm...We believe we will all find a solution together peacefully," he told reporters following Thursday\'s meeting.\n\nProtest leader Jagjit Singh Dallewal also told reporters the farmers would hold off their march for now.\n\n"When the meetings have started, if we move forward (towards Delhi) then how will meetings happen?" Dallewal told reporters, adding that the protest "will continue peacefully".\n\nThousands of farmers had embarked on the "Delhi Chalo", or "Let\'s go to Delhi" march earlier this week to press the government to set a minimum price for their produce, but they were stopped by security forces about 200 kms (125 miles) away from the capital, triggering clashes.\n\nThe protests erupted a few months before India is due to hold national elections in which Prime Minister Narendra Modi is seeking a third term. Farmers form an influential voting bloc.\n\nThe farmers remained camped on the border between Punjab and Haryana states on Friday. Security forces have used concrete and metal barricades, as well as drones carrying tear gascanisters, to stop them for advancing.\n\nThe protest comes two years after Modi\'s government, following a similar protest movement, repealed some farm laws and promised to find ways to ensure support prices for all produce.\n\n(Writing by Sakshi Dayal; editing by Miral Fahmy)',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Triplet

• Dataset: test
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

processed_yahoo_finance_stockmarket_news

• Dataset: processed_yahoo_finance_stockmarket_news at 6949dc8
• Size: 14,845 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 7 tokens</li><li>mean: 16.02 tokens</li><li>max: 42 tokens</li></ul>	<ul><li>min: 10 tokens</li><li>mean: 788.29 tokens</li><li>max: 8192 tokens</li></ul>	<ul><li>min: 104 tokens</li><li>mean: 270.14 tokens</li><li>max: 669 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>If You'd Invested $1,000 in Amazon Stock 10 Years Ago, Here's How Much You'd Have Today</code>	<code>It might be hard to imagine, butAmazon(NASDAQ: AMZN)began only 30 years ago as an online bookseller. Now the site sells a wide range of goods. The business has grown quickly and includes the popular Amazon Prime subscription service, electronic devices, and Amazon Web Services (AWS).The shares have also generated a lot of wealth for investors. Investors who bought shares a decade ago, and held them, would have seen an increase of almost 1,000%.Trouncing the marketAmazon had itsinitial public offering (IPO)in 1997, when its sales were about $148 million for the year. The figure grew to nearly $575 billion last year.But you didn't have to buy the shares at the IPO price to make a lot of money. Over the last decade, Amazon's shares have appreciated 945%. That easily bested theS&P 500's 227% total return.Even starting with a relatively small $1,000 just 10 years ago, you would now have about $10,500. Placing the same amount in the S&P 500 would've resulted in about $3,300.Amazon's stock wi...</code>	<code>My husband of almost a year went home with another girl while we were freshly split up. Nothing happened, but he still went home with her from a bar and slept over in her bed. I am pissed about that, and so we split then. He since came back to me apologizing..begging for another chance...and I gave him one. Things went well for a couple of weeks, but I got drunk one night and brought it up, so we fought about it. Then at random times I think about it and bring it up to him. He gets so angry every time it comes up and has asked if he has to deal with hearing about this every single day. He asked me to decide if I could stay with him and deal with this or not. He took all of his stuff out of my place (we are separated, but he stays with me since we are working things out) and is staying in his place because we argued. We talked and I thought it was a good talk. I usually blame everything on him,but today I took some blame and truly want to work it out.He just got frustrated,what to do</code>
  <code>Chemicals Company Dow Cuts Outlook Amid Production Challenges</code>	<code>Chemicals Company Dow Cuts Outlook Amid Production ChallengesDow Inc.(NYSE:DOW) shares are trading lower after it revised its third-quarter outlook.The company lowered its guidance for revenue to around $10.6 billion (vs. consensus $11.0 billion) from $11.1 billionearlierandexpects operating EBITDA of around $1.3 billion.The revised outlook reflects a major unplanned event at a Texas ethylene cracker in late July, alongside higher input costs and margin compression in Europe.However, these challenges are expected to be partly mitigated by improved pricing and feedstock costs in North America for Packaging & Specialty Plastics.Also Read:Linde Invests Over $2B For Clean Hydrogen, Inks Supply Deal With Dow: DetailsJim Fitterling, chair and chief executive officer, said, “As we look to the fourth quarter, we expect typical seasonality in demand. However, we expect a positive impact from lower turnaround costs, higher operating rates as we ramp up our Texas cracker, and fewer weather-relate...</code>	<code>I have a fourth degree polynomial, namely:\n\ny = 0.0003x4 - 0.0272x3 + 0.6806x2 - 0.8345x + 0.5659\n\nWhen plotting this line, from points x=560 to x=790, the way that that curve looks...I want to know at what point am i going to get the greatest increase in y, with the least increase in x...do you get waht i'm saying? At what point, like at say 700 to 701, is that where i am going to see the largest y increase with the least x increase. I want it to be proportaional,like for example:\n\nMaybe from 700-701 is just an increase of 2 in the y direction, but if going from 703-705 i get an incrase of 5, thats more benificial for me, cause i more than doubled my input...get waht i'm saying??\n\nPlease help here, i'm not sure if i should be taking the derivitive or what.\n\nThanks</code>
  <code>Beyond Meat Stock Is Down 20% This Year, Time to Buy?</code>	<code>Beyond Meat Inc.(NASDAQ: BYND)continues to struggle with its mission of delivering "the future of protein" amid weak sales and large recurring losses. That trend continued as the plant-based meat leader posted its latest earnings report on May 8.The market remains skeptical that the company can manage a growth rebound. The stock is now down more than 40% over the past year.Is now the time to buy Beyond Meat, or are investors better off leaving this one off their plate?A recap of Beyond Meat's Q1 earnings reportFor the first quarter of 2024, Beyond Meat's revenue fell by 18% year over year to $75.6 million, marking the eighth consecutive quarter in decline.The top-line weakness was driven by a 16.1% decrease in the volume of products sold despite an effort to stir up demand through promotional discounting. The company also cited the impact of discontinuing its "Beyond Meat Jerky" along with lower "Beyond Chicken" sales, against large initial orders from Europe in early 2023.Demand was s...</code>	<code>ASTOR FAMILY, a famous American<br>family representing one of the three or four<br>greatest private properties in the world. A<br>family in the Old World sense, — a territorial<br>aristocracy, impossible to destroy, and fortified<br>with legal immunities and privileges, — can<br>hardly be founded in America; but the Astors<br>have approached it as nearly as our institutions<br>will admit. They form a group of immense<br>hereditary real-estate owners, with holdings so<br>solidly based and well distributed in the metropolis<br>of America that no apparent catastrophe<br>save a failure of heirs could extinguish it; and<br>though originally springing from mercantile<br>business, removed by some three-quarters of a<br>century from its actual conduct. For many years<br>they were known as “the landlords of New<br>York,” and the best of landlords, prompt, just and<br>courteous; they still probably form the largest<br>set of individual real-estate holders. The<br>family is also connected with notable municipal<br>charities and public foundations. See<br> (1763...</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

processed_yahoo_finance_stockmarket_news

• Dataset: processed_yahoo_finance_stockmarket_news at 6949dc8
• Size: 14,845 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 7 tokens</li><li>mean: 16.07 tokens</li><li>max: 32 tokens</li></ul>	<ul><li>min: 11 tokens</li><li>mean: 789.07 tokens</li><li>max: 7142 tokens</li></ul>	<ul><li>min: 101 tokens</li><li>mean: 269.38 tokens</li><li>max: 711 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>What the Tech Bubble in 2000 May Tell Us About the Stock Market Today</code>	<code>Spencer Platt / Getty ImagesKey TakeawaysRecent losses for tech stocks have coincided with gains for utilities, healthcare, and consumer staples, an echo of the 2000 dotcom bubble, according to a recent Deutsche Bank note.Uncertainty about the U.S. election and monetary policy could continue to weigh on tech stocks in the coming months, according to one analyst.Even with the bursting of the dotcom bubble, investments in tech have significantly outperformed the broader market over the long-term.There’s reason to believe history is repeating itself, according to a recent Deutsche Bank note that draws parallels between the current rotation out of tech mega-caps and the bursting of the tech bubble in 2000.In the 9 months after the bubble reached its peak in March 2000, tech stocks fell more than 50%. The Consumer Staples, Utilities, and Healthcare sectors, meanwhile, each rose by more than 35%.Deutsche BankResearch analyst Jim Reid sees similarities between that moment and thecurrent secto...</code>	<code>My friend has a 1 year old girl and her husband cheated on her with the receptionist at the doctors office they take the little girl. They are now divorced. He has 1 over night visit per week sometimes her show sometimes he doesn't. And when he does he often times drops her off at his mothers or drags the kid from girl friend to girl friends house. How do you convince him that his kid should come first. Her mom is a wonderful mother and person that puts her first. The way he treats her kid has her in turmoil. Please help.</code>
  <code>Europe’s stock leaders are fading in bad sign for future returns</code>	<code>(Bloomberg) — The engines behind two years of European stock gains are losing power, leaving the region’s equities facing a void at a time when concerns over slowing growth and China tensions are testing investor confidence.Most Read from BloombergHousing’s Worst Crisis in Decades Reverberates Through 2024 RaceAn Affordable Nomadic Home Design Struggles to Adapt to Urban LifeUS Driving and Congestion Rates Are Higher Than EverA City Finds Success Using 'Trees as Medicine'The Hague Is World’s First City to Ban Oil and Air Travel AdsA luxury sector led by LVMH Moët Hennessy Louis Vuitton SE (MC.PA,LVMHF) has tumbled over the past six months along with automotive firms, while in more recent months healthcare heavyweights such as Novo Nordisk A/S (NVO,NOVO-B.CO) and tech leaders including ASML Holding NV (ASML) have slid from their peaks. And with no obvious candidates to take the baton, the region’s equity performance has been left looking exposed.Already this year, investors have withdra...</code>	<code>I'm quite conservative for someone my age. I don't like heavy drinking (I do drink. I just don't get wasted) and I don't like wild parties. I'm very religious and I'm looking for someone who shares my convictions who I will eventually hopefully marry. The problem that I seem to have is most people my age or either non-religious partiers or their fundamentalist Bible beaters who don't know how to do much other than quote scriptures. I've tried online dating and I've tried asking out people at church. Why can't I find a normal person that shares my beliefs? The ones I do find tend to always dating someone else before I ever get a chance.</code>
  <code>2 Billionaire Investors Are Selling Nvidia and Buying This Artificial Intelligence (AI) Stock Instead</code>	<code>Great investing minds appear to be thinking alike about two top AI stocks.</code>	<code>"A Song" (Once—and only once—you gave), a poem by Edwin Arnold<br>"A Song" (Gentle nymphs, be not refusing), a poem by William Browne<br>"A Song" (The sparkling eye, the mantling cheek), a poem by William Cowper<br>"A Song" (On a summer's day as I sat by a stream), a poem by Paul Laurence Dunbar<br>"A Song" (Thou art the soul of a summer's day), a poem by Paul Laurence Dunbar<br>"A Song" (My heart has flown on wings to you, away), a poem by Francis Ledwidge<br>"A Song" (Yes! "lower to the level"), a poem by Frances Sargent Osgood<br>"A Song" (Call me pet names, dearest! Call me a bird), a poem by Frances Sargent Osgood<br>"A Song" (O, red is the English rose), a poem by Charles Alexander Richmond<br>"A Song" (I thought no more was needed), a poem by William Butler Yeats<br><br>See also<br> Song<br> The Song</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->