finpeak-ai

Financial SLM - Model Card

Model Details

• Model Name: Financial Small Language Model (SLM)
• Model Type: GPT-2 Fine-tuned Transformer
• Parameters: 95.50M
• Architecture: 8 layers, 12 attention heads, 768 embedding dim
• Vocabulary: GPT-2 (50,257 tokens)
• Training Date: November 15, 2025

Training Approach

This model was trained using a two-phase curriculum learning approach:

Phase 1: General English Language Learning

• Dataset: OpenWebText (general English text)
• Purpose: Learn fundamental language structure, grammar, and syntax
• Iterations: 30,000
• Final Loss: 8.1097
• Outcome: Model gains understanding of English before specializing

Phase 2: Financial Domain Specialization

• Datasets: Finance Alpaca + Financial PhraseBank + Custom Investment Data
• Purpose: Specialize in financial concepts, terminology, and analysis
• Iterations: 25,000
• Best Validation Loss: 4.9894
• Outcome: Model combines English fluency with financial expertise

Training Data

Phase 1: General English (OpenWebText)

• General-purpose English text corpus
• Tokens: N/A
• Purpose: Foundation language understanding

Phase 2: Financial Domain

• Finance Alpaca: 68,912 financial Q&A examples
• Financial PhraseBank: 3,100 sentiment analysis examples
• Custom Investment Data: 100 curated investment examples
• Total Training Examples: 82,900
• Total Tokens: 11,490,898

Training Configuration

Phase 1 Configuration

• Optimizer: AdamW (lr=1e-07, weight_decay=0.1)
• Scheduler: Linear warmup (6,000 steps) + Cosine annealing
• Iterations: 30,000

Phase 2 Configuration

• Optimizer: AdamW (lr=5e-06, weight_decay=0.1)
• Batch Size: 8 (effective: 32)
• Training Steps: 25,000
• Learning Rate Schedule: Linear warmup (2,500 steps) + Cosine annealing
• Gradient Clipping: max_norm=1.0
• Mixed Precision: bfloat16
• Hardware: GPU (CUDA)

Performance

• Phase 1 Final Loss: 8.1097
• Phase 2 Best Validation Loss: 4.9894
• Phase 2 Final Train Loss: 4.9975
• Phase 2 Final Val Loss: 5.0073
• Total Evaluations: 50

Capabilities

• Financial question answering
• Investment analysis and recommendations
• Sentiment analysis of financial statements
• Company financial health assessment
• Portfolio diversification advice
• Financial concept explanations
• Market trend interpretation
• Risk assessment guidance

Limitations

• Educational purposes only - not financial advice
• May generate incorrect or outdated information
• Should not be sole basis for investment decisions
• Requires human verification and professional advice
• Training data cutoff may not reflect latest market conditions
• Small model size may limit reasoning capabilities

Intended Use

• Financial education and learning
• Investment research assistance
• Financial literacy improvement
• Prototype for financial chatbots
• Research on domain adaptation techniques

Ethical Considerations

• Always include disclaimers about not being financial advice
• Users should consult licensed financial advisors
• Model outputs should be fact-checked
• Not suitable for automated trading systems
• May reflect biases present in training data
• Should not be used for high-stakes financial decisions

Citation

If you use this model, please cite: Financial SLM (2025) by Satgun Singh Sodhi 95.5M parameter GPT-2 transformer with two-phase training Phase 1: General English (OpenWebText) Phase 2: Financial domain (Finance Alpaca + Financial PhraseBank + Custom data)