financial-sentiment-model

Financial Sentiment Analysis using XLM-RoBERTa Overview

This project implements a multiclass financial sentiment analysis system using Natural Language Processing (NLP) and transformer-based deep learning models. A pre-trained XLM-RoBERTa model is fine-tuned to classify financial text into four sentiment categories: Neutral, Bullish, Bearish, and Strongly Bullish.

The training pipeline follows robust machine learning practices such as stratified cross-validation, two-phase fine-tuning, class imbalance handling, resumable training, and comprehensive evaluation. The final model is saved in a Hugging Face–compatible format for easy reuse and deployment.

Problem Statement

Financial text data such as tweets, news headlines, and market commentary often contains implicit sentiment that can influence investment decisions. Accurately classifying this sentiment is challenging due to domain-specific language, class imbalance, and subtle contextual cues. This project addresses these challenges by adapting a large pre-trained language model to the financial sentiment classification task.

Model Description

Base model: xlm-roberta-base

Model type: Transformer-based sequence classification

Framework: TensorFlow (TF-Keras)

Task: Multiclass text classification

Number of labels: 4

XLM-RoBERTa is a multilingual transformer model trained on large-scale textual data. Fine-tuning allows the model to adapt its learned language representations specifically to financial sentiment.

Fine-Tuning Strategy

The model is trained using a two-phase fine-tuning approach.

Phase 1 trains only the classification head while freezing the transformer layers. This allows the model to quickly adapt to the target labels without disturbing the pre-trained representations.

Phase 2 unfreezes the entire model and continues training with a smaller learning rate. This enables deeper adaptation of the language representations to the financial domain while maintaining training stability.

This approach improves convergence, reduces overfitting, and leads to better generalization.

Dataset Construction

The training dataset is created by combining local sentence-level financial datasets and publicly available Hugging Face datasets. All datasets are mapped to a unified label schema to ensure consistency.

Local datasets include sentence files grouped by agreement levels and sentiment polarity. Online datasets include financial tweets and news sentiment data sourced from Hugging Face. All samples are cleaned, merged, and converted into a single dataset for training.

Label Mapping

The classification task uses the following label structure:

0: Neutral

1: Bullish

2: Bearish

3: Strongly Bullish

This mapping is applied consistently across all datasets and during model training and evaluation.

Class Imbalance Handling

Financial sentiment datasets often exhibit class imbalance. To mitigate this issue, class weights are computed using a balanced weighting strategy and applied during training. This ensures that minority classes contribute proportionally to the loss function and improves overall model fairness.

Cross-Validation

The model is evaluated using stratified k-fold cross-validation with five folds. Stratification preserves the original class distribution in each fold, providing a more reliable and unbiased evaluation of model performance.

Each fold undergoes the complete two-phase training process, and validation results are recorded separately.

Resume-Safe Training

To support long-running training and recover from interruptions, the training pipeline saves progress after each completed fold in a JSON-based state file. This allows training to resume from the last completed fold without repeating previous work.

Evaluation Metrics

After training across all folds, predictions from each validation set are aggregated. The model is evaluated using:

Accuracy

Precision

Recall

F1-score

Confusion matrix

A detailed classification report and visual confusion matrix are generated and saved for analysis.

Model Selection and Saving

The best-performing model across all folds is selected based on validation accuracy. The corresponding model weights and tokenizer are saved in a Hugging Face–compatible format, making the model ready for inference, deployment, or further fine-tuning.

Inference Example from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("path_to_model") model = TFAutoModelForSequenceClassification.from_pretrained("path_to_model")

text = "The company is expected to report strong quarterly earnings" inputs = tokenizer(text, return_tensors="tf") outputs = model(**inputs) predicted_label = outputs.logits.argmax(axis=1)

Applications

This model can be applied to a wide range of financial NLP tasks, including sentiment analysis of market news, stock-related social media monitoring, trading decision support systems, and academic research in financial text analytics.

Conclusion

This project demonstrates a complete and reliable NLP pipeline for financial sentiment analysis using transformer fine-tuning. By combining robust data preparation, careful training strategies, and thorough evaluation, the resulting model achieves strong generalization performance and is suitable for real-world financial text analysis tasks.

License

This project is intended for research and educational use. Users should verify the licenses of individual datasets before using the model in commercial applications.