openmedallion-finsentiment

OpenMedallion-FinSentiment

Financial Sentiment Classification via QLoRA Fine-Tuning

⚠️ DISCLAIMER: This repository contains training infrastructure and prepared data for financial sentiment analysis. No trained model weights are published — training requires a GPU (≥12GB VRAM). Sentiment predictions do NOT constitute financial advice.

Repository Contents

Quick Start

Prerequisites

pip install torch transformers datasets peft bitsandbytes accelerate

Train

python scripts/fine_tune_qwen.py \
    --dataset-dir data/sentiment \
    --output-dir ./outputs \
    --model-name Qwen/Qwen2.5-7B-Instruct \
    --epochs 3 \
    --batch-size 4 \
    --gradient-accum 4

Push to this Hub repo after training:

HF_TOKEN=hf_... python scripts/fine_tune_qwen.py \
    --dataset-dir data/sentiment \
    --output-dir ./outputs \
    --push_to_hub \
    --hub_username oyi77 \
    --hub_repo_name openmedallion-finsentiment

Evaluate

python scripts/eval_finsentiment.py \
    --model-path ./outputs/final_model

Training Data

• Source: 21,884 financial headlines with sentiment labels extracted from OpenMedallion instruction-tuning data
• Format: JSONL with {messages, label} structure
• Split: 90/10 train/validation (applied during training)
• Labels: positive (0), negative (1), neutral (2)
• Domain: Primarily gold-commodity headlines (price movements, imports, market outlook)

Data Format

{"messages": [{"role": "user", "content": "gold rate today: track gold price in india and market outlook"}], "label": "positive"}

The preprocess_function wraps user messages with the instruction prefix "Classify the sentiment of this financial text:\n\n" to match inference-time formatting.

Model Architecture

Base Model

• Foundation: Qwen/Qwen2.5-7B-Instruct
• Parameters: 7 billion (4-bit quantized via bitsandbytes)
• Context Length: 32,768 tokens
• Fine-Tuning Method: QLoRA (Quantized Low-Rank Adaptation)

QLoRA Configuration

from peft import LoraConfig
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

lora_config = LoraConfig(
    r=16,                     # Rank
    lora_alpha=32,            # Scaling factor
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Training Hyperparameters

• Epochs: 3
• Batch Size: 4 (per device) × 4 (gradient accumulation) = 16 effective
• Learning Rate: 2e-4 with cosine schedule
• Optimizer: AdamW (8-bit via bitsandbytes)
• Mixed Precision: bfloat16
• Hardware: NVIDIA GPU with ≥12GB VRAM (RTX 3060 or better)
• Training Time: ~2-4 hours

Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    load_in_4bit=True,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "./outputs")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

def predict_sentiment(text: str) -> str:
    messages = [
        {"role": "system", "content": "You are a financial sentiment analysis assistant."},
        {"role": "user", "content": f"Classify the sentiment of this financial text:\n\n{text}"}
    ]
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=10, temperature=0.1)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("assistant")[-1].strip().lower()

print(predict_sentiment("Apple reports record Q4 earnings"))  # "positive"

Limitations

1. Not Yet Trained: This repo contains training infrastructure and data only. Run training yourself to get the model.
2. Gold Bias: Training data is primarily gold-commodity headlines; financial sentiment across other sectors may underperform.
3. Heuristic Labels: Labels extracted via keyword pattern matching, not human annotation.
4. Classification Only: Outputs three classes (positive/negative/neutral), not continuous sentiment scores.

Citation

@misc{openmedallion-finsentiment-2026,
  author = {oyi77},
  title = {OpenMedallion-FinSentiment: Financial Sentiment Classification via QLoRA},
  year = {2026},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/oyi77/openmedallion-finsentiment}}
}

@article{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2412.15115},
  year={2024}
}

@article{dettmers2024qlora,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal={NeurIPS},
  year={2023}
}

Contact

• Repository: https://huggingface.co/oyi77/openmedallion-finsentiment
• Dataset: https://huggingface.co/datasets/oyi77/OpenMedallion
• Main Repo: https://github.com/oyi77/OpenMedallion

---

Last Updated: 2026-07-21