modernbert-fingpt

ModernBERT Fine-tuned for Financial Text Sentiment Analysis

This project fine-tunes the ModernBERT model on the FinGPT sentiment dataset for financial text sentiment analysis.

Dataset & Model

• Model: answerdotai/ModernBERT-base
• Dataset: FinGPT/fingpt-sentiment-train
• Task: Multi-class sentiment classification (9 categories)
• Domain: Financial text analysis

ModernBert

ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. It leverages architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, Local-Global Alternating Attention for efficiency on long inputs, Unpadding and Flash Attention for efficient inference.

FinGPT Sentiment Analysis Dataset

Contains 76,772 test rows (17,919,695 tokens)

Sentiment Categories

The model classifies text into 9 fine-grained sentiment levels:

Model Configuration

Parameters

• Max Sequence Length: 512 tokens
• Batch Size: 16
• Learning Rate: 2e-5 with warmup
• Epochs: 3 with early stopping
• Optimizer: AdamW with weight decay (0.01)

Features

• Early Stopping: Prevents overfitting (patience=3)
• Best Model Loading: Automatically loads best checkpoint
• Mixed Precision: FP16 training for speed optimization
• Stratified Splitting: 80/20 train/validation split

Evaluation Metrics

• Accuracy: Overall classification accuracy
• F1-Score: Weighted F1-score across all classes
• Precision: Weighted precision
• Recall: Weighted recall
• Confusion Matrix: Visual analysis of classification performance
• Classification Report: Detailed per-class metrics

Performance

Training Time (on T4 GPU)

• Total Training: ~30-45 minutes
• Per Epoch: ~10-15 minutes
• Evaluation: ~2-3 minutes

Training Results (Actual)

• Loss: 0.3741
• Accuracy: 0.9043
• F1: 0.9026
• Precision: 0.9022
• Recall: 0.9043

Deployment Options

• API Deployment: Create REST API using FastAPI
• Batch Processing: Set up automated sentiment analysis pipeline
• Real-time Analysis: Integrate with financial data streams

References

• ModernBERT Paper
• FinGPT Project
• Hugging Face Transformers
• Financial Sentiment Analysis Survey