finetuned-tinyllama-finmarba

Model Card for seifelzeiny/finetuned-tinyllama-finmarba

Model Description

TinyLlama fine-tuned to answer financial news headlines sentiment Q&A, using the FinMarBa dataset.
This model predicts sentiment (Negative/Neutral) from financial headlines in English via causal language modeling.

• Developed by: Seifeldin Elzeiny (@seifelzeiny)
• Model type: Causal language model (LLM, LoRA/PEFT)
• Language(s): English
• License: MIT
• Finetuned from: TinyLlama/TinyLlama-1.1B-Chat-v0.4

Model Sources

• Repository: seifelzeiny/finetuned-tinyllama-finmarba
• Dataset: baptle/financial_headlines_market_based

Uses

Direct Use

Sentiment Q&A for financial headlines:

• Use in financial news analytics tools
• Educational examples for stock sentiment
• Research on financial text understanding by small language models

Downstream Use

Can be extended for broader financial NLP tasks:

• Financial chatbot assistants
• Automated news classification
• Sentiment-based trading signals (research only)

Out-of-Scope Use

• Do not use for medical, legal, or high-stakes banking advice.
• Not suitable for real-time trading decisions without thorough validation.
• May not generalize to non-financial or multi-lingual content.

Bias, Risks, and Limitations

• Model trained only on English headlines, mostly US/EU market news.
• May reflect dataset biases (e.g., over/under-representation of neutral or negative market moves).
• Performance in non-news or conversational contexts is untested.

Recommendations

For academic or research use.
Validate outputs before applying to production finance systems.
Future work should expand to multi-class and multi-lingual sentiment coverage.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("seifelzeiny/finetuned-tinyllama-finmarba")
tokenizer = AutoTokenizer.from_pretrained("seifelzeiny/finetuned-tinyllama-finmarba")

prompt = "### Question: What is the sentiment of this news?\nFederal Reserve raises rates.\n### Answer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• Dataset: baptle/financial_headlines_market_based
• Filtered: Only samples with valid sentiment labels (0 = Negative, 1 = Neutral; excluded -1 Unknown).
• Size: About 38,000 labeled headlines.

Training Procedure

• Preprocessing: Filtered out unknown (-1) labels, mapped headlines and sentiments to Q&A format.
• Hyperparameters: 3 epochs, batch size 2, LoRA r=8, fp16.
• Validation: 80/20 train/validation split; monitored train+validation loss.

Evaluation

Metrics

• Training loss: ~0.89 (last epoch)
• Validation loss: ~0.93 (last epoch)

Results

Model achieves near-equal training and validation loss, indicating good generalization without overfitting (see loss table above).
Direct accuracy scores are not used due to causal LM format, but manual validation shows plausible sentiment answers.

Environmental Impact

• Hardware Type: Nvidia A10G GPU (Modal cloud), or local CUDA device
• Training time: ~4 minutes/epoch x 3 epochs
• Carbon Emitted: Estimated <0.2kg CO₂ (low-power cloud GPU)

Technical Specifications

• Architecture: TinyLlama 1.1B Chat
• Training: LoRA/PEFT, HuggingFace Trainer
• Compute: Modal cloud GPU, PyTorch+Accelerate

Citation

@misc{seifelzeiny2025finmarba, author = {Seif Basha}, title = {Finetuned TinyLlama for Financial News Sentiment Q&A (FinMarBa)}, year = {2025}, url = {https://huggingface.co/seifelzeiny/finetuned-tinyllama-finmarba}, note = {Financial NLP, LoRA, FinMarBa, TinyLlama} }

Model Card Contact

• Seifeldin Elzeiny (@seifelzeiny) – [email protected]