sp500-predictor-individual-headlines

<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>

SP500-Predictor-Individual-Headlines

This model is a fine-tuned version of distilbert/distilroberta-base, trained on a financial news headlines dataset from Kaggle. The headlines were assigned a classification based on the movement of the S&P 500 index on the next trading day following the publication of each headline. This movement classification was determined using S&P 500 closing prices obtained separately from Yahoo Finance and is defined as follows:

• Increase: If the closing price increased by more than 0.5%
• Decrease: If the closing price decreased by more than 0.5%
• Maintain: If the closing price stayed within -0.5% to 0.5%

It achieves the following results on the evaluation set:

• Loss: 1.0498
• Accuracy: 0.4765

Base Model description

This model is based on the distilroberta-base architecture, a distilled version of RoBERTa-base. DistilRoBERTa retains 97% of the language understanding capabilities of RoBERTa-base while being twice as fast. It has 6 layers, 768 dimensions, and 12 attention heads, totaling 82 million parameters (compared to 125 million for RoBERTa-base). The model is case-sensitive and distinguishes between different cases in text.

Intended uses & limitations

The goal of this model is to predict the movement of the S&P 500 index based on financial news headlines. However, this initial version is not intended for production use. While it may provide insights into the potential impact of news on market movements, fine-tuning on more specific and extensive datasets will be necessary to improve its accuracy and reliability.

Training and evaluation data

The training data consists of financial news headlines sourced from Kaggle. The headlines were classified based on the movement of the S&P 500 index on the following trading day, with closing prices gathered separately from Yahoo Finance. Movements were classified into three categories: increase, decrease, or maintain, based on the criteria mentioned above.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Framework versions

• Transformers 4.42.4
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1