finlora-adapter

FinLoRA — Financial Instruction LoRA Adapter

LoRA adapter fine-tuned on SmolLM2-135M-Instruct for financial instruction following. Trained on financial Q&A/instruction data (DeividasM/financial-instruction-aq22) with PEFT (rank 16, alpha 32, dropout 0.05 on q/k/v/o projections).

Training note: this adapter was trained on CPU as an experiment; the evaluation below was run on a Kaggle Tesla T4 (fp16).

Evaluation — base vs adapter (honest, reproducible)

Held-out 10% of the aq22 instruction set (120 samples, seed 42), ROUGE scoring, max 96 new tokens.

Verdict: small but real gains on ROUGE-1 (+0.005) and ROUGE-L (+0.018); a slight ROUGE-2 regression (−0.012) — consistent with the small 135M base and CPU training budget. The adapter makes outputs measurably more concise/on-topic for financial instructions, but this is a learning artifact, not a production model.

Side-by-side generations (held-out)

1. You are a financial analyst categorizing tweets into specific financial topics. Given a tw...

• Reference: Stock Commentary
• Base: The topic name for this tweet is "TIL Cell Therapy Set To Prove Itself."
• Tuned: Stock Market 2. You are a financial sentiment analysis expert. Your task is to analyze the sentiment expre...
• Reference: positive
• Base: Positive
• Tuned: Positive 3. You are a financial analyst identifying sentiment towards specific entities in financial n...
• Reference: The sentiment in this text about McDonald is positive. The sentiment in this tex
• Base: McDonald is a positive sentiment.

McDonald is a positive sentiment because it indicates that the company is i

• Tuned: McDonald is a positive sentiment.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = PeftModel.from_pretrained(base, "vivekkopthsd/finlora-adapter")

Limitations

• 135M base — bounded reasoning capacity
• Mixed eval deltas (ROUGE-2 regression) — verify per use case
• Single-domain financial instruction data; not safety-tuned for general chat