finetuned-SmolLM2-1.7B-finmarba

Model Card for seifelzeiny/finetuned-SmolLM2-1.7B-finmarba

This model is a parameter-efficient, LoRA-finetuned version of SmolLM2-1.7B, specialized for financial news sentiment classification (distinguishing "Negative" vs. "Neutral" headlines).

Model Details

Model Description

SmolLM2-1.7B fine-tuned to answer financial news headlines sentiment Q&A, using the FinMarBa dataset.
This model predicts sentiment (Negative/Neutral) from financial headlines in English via causal language modeling with LoRA.

• Developed by: Seifeldin Elzeiny (@seifelzeiny)
• Model type: Causal language model (LLM, LoRA/PEFT)
• Language(s): English
• License: MIT
• Finetuned from: HuggingFaceTB/SmolLM2-1.7B

Model Sources

• Repository: https://huggingface.co/seifelzeiny/finetuned-SmolLM2-1.7B-finmarba
• Dataset: baptle/financial_headlines_market_based
• Demo: [Try it out NOW!]

Uses

Direct Use

• Provide a financial news headline to get sentiment prediction (Negative/Neutral).
• Useful for financial analysts, fintech NLP, and business analytics.

Downstream Use

• Can be integrated into custom pipelines, dashboards, or chatbots requiring financial sentiment classification.

Out-of-Scope Use

• Not suitable for non-financial text, other languages, or nuanced sentiment beyond "Negative/Neutral."

Bias, Risks, and Limitations

• May reflect bias from training data (e.g., particular institutions or event types).
• Only trained on English financial news; unpredictable on other text domains.

Recommendations

Users should verify outputs before acting, especially in critical or high-stakes analysis settings.

How to Get Started with the Model

Try in Hugging Face Widget

1. Go to the model page and use the Hosted Inference widget.
2. Paste a headline like:
   Apple profits drop unexpectedly in Q3.
3. Receive a sentiment prediction below.

Use in Python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "seifelzeiny/finetuned-SmolLM2-1.7B-finmarba"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "### Question: What is the sentiment of this news?\nApple profits fall in Q3.\n### Answer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs, skip_special_tokens=True))

Sample Questions

1. Tesla stock surges after record vehicle deliveries in Q2.

2. Global markets tumble amid renewed recession fears.

3. Major auto manufacturer announces massive recall after defects found.

4. Oil prices remain stable despite geopolitical tensions.

5. Amazon to invest $10 billion in new US warehouses.

Training Details

Training Data

• Based on [baptle/financial_headlines_market_based], filtered for English and class labels 0 ("Negative") and 1 ("Neutral").

Training Procedure

Preprocessing

• Headlines formatted as prompts:
  ### Question: What is the sentiment of this news?\n{headline}\n### Answer: {label}

Training Hyperparameters

• LoRA r=16, alpha=32, dropout=0.1 (targeting QKV/O projections)
• 5 epochs, batch size 4 (gradient accumulation 2)
• Adam optimizer, fp16 mixed precision (PyTorch/Transformers)

Speeds, Sizes, Times

• Total steps: 2835
• Final training loss: ~0.94
• Validation loss: ~1.06

Evaluation

Testing Data, Factors & Metrics

• Validation split from the above dataset, stratified random split.
• Main metric: validation loss (cross-entropy).
• For qualitative accuracy: answer plausibility on held-out headlines.

Results

Model achieves near-equal training and validation loss, indicating good generalization without overfitting (see training logs).
Manual validation shows plausible sentiment answers (“Negative”, “Neutral”) for held-out headlines.

Summary

The model can reliably classify headline sentiment as Negative or Neutral.

Technical Specifications

Model Architecture and Objective

• SmolLM2-1.7B, Llama-based, fine-tuned with LoRA for sequence classification via language modeling.

Environmental Impact

• Hardware Type: Nvidia A10G GPU or equivalent
• Training time: ~8 minutes/epoch x 5 epochs
• Carbon Emitted: Estimated <0.3kg CO₂

Technical Specifications

• Architecture: SmolLM2-1.7B
• Training: LoRA/PEFT, HuggingFace Trainer
• PEFT config: r=16, lora_alpha=32, lora_dropout=0.1
• Compute: Modal cloud GPU, PyTorch+Accelerate

Citation

@misc{seifelzeiny2025finmarba_smollm2, author = {Seifeldin Elzeiny}, title = {Finetuned SmolLM2-1.7B for Financial News Sentiment Q&A (FinMarBa)}, year = {2025}, url = {https://huggingface.co/seifelzeiny/finetuned-SmolLM2-1.7B-finmarba}, note = {Financial NLP, LoRA, FinMarBa, SmolLM2} }

More Information

Contact author for questions or collaboration. [[email protected]]