serbfin-sentiment-llama-3.2-3b

SerbFin-Sentiment -- Llama 3.2 3B (LoRA adapter)

Built with Llama. QLoRA (4-bit NF4) fine-tuned adapter for meta-llama/Llama-3.2-3B-Instruct, for 3-class Serbian financial sentiment classification (positive / negative / neutral) on short financial news headlines, judged from an investor / market perspective (not emotional sentiment).

First publicly available Serbian-language financial sentiment model.

Results

Evaluated on a 240-headline human-labeled gold test set (never used for training), same fixed prompt at zero-shot baseline / training / eval time. Full per-class breakdown and confusion matrices: reports/eval.md in the GitHub repo.

Fine-tuned macro-F1 beats both required baselines (zero-shot Llama 3.2 3B, and a translate-to-English + ProsusAI/finbert pipeline).

Training data

filipovicfilip222/serbfin-sentiment -- train + validation splits (silver-labeled by Claude Haiku). The test split used for the numbers above is human-gold-labeled and was never seen during training.

Hyperparameters

QLoRA (configs/lora.yaml):

r: 16
lora_alpha: 32
lora_dropout: 0.05
bias: none
task_type: CAUSAL_LM
target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj

Training (configs/train.yaml):

output_dir: outputs/serbfin-llama-3.2-3b
num_train_epochs: 3
per_device_train_batch_size: 8
gradient_accumulation_steps: 2
learning_rate: 0.0002
lr_scheduler_type: cosine
warmup_ratio: 0.03
optim: paged_adamw_8bit
bf16: true
gradient_checkpointing: true
max_length: 512
logging_steps: 10
eval_strategy: steps
eval_steps: 50
save_strategy: steps
save_steps: 50
load_best_model_at_end: true
metric_for_best_model: eval_loss
seed: 42
completion_only_loss: true
report_to: none

Prompt template

Identical at train / zero-shot baseline / eval time (src/label/silver.py::SYSTEM_PROMPT_V1):

<system>
Ti si klasifikator finansijskog sentimenta. Za dati naslov vesti odredi sentiment sa
perspektive investitora.
Odgovori isključivo jednom rečju: positive, negative ili neutral.
</system>
<user>
Naslov: {text}
Sentiment:
</user>
<assistant>
{label}

How to use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model = "meta-llama/Llama-3.2-3B-Instruct"
adapter_id = "filipovicfilip222/serbfin-sentiment-llama-3.2-3b"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_model, quantization_config=bnb_config, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

messages = [
    {"role": "system", "content": "Ti si klasifikator finansijskog sentimenta. Za dati naslov vesti odredi sentiment sa perspektive investitora.\nOdgovori isključivo jednom rečju: positive, negative ili neutral."},
    {"role": "user", "content": "Naslov: NIS objavio rekordnu dobit u prvom kvartalu\nSentiment:"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Reproduction

git clone https://github.com/filipovicfilip222-alt/SerbFin-Sentiment
cd SerbFin-Sentiment && pip install -e .
python -m src.data.split --train-val-only
python -m src.train.qlora_train      # requires a CUDA GPU
python -m src.eval.finetuned_eval

Intended use & limitations

Financial-news-headline sentiment, Serbian language, investor/market framing. Not intended for emotional sentiment, social media text, or languages other than Serbian. The positive/neutral boundary is the most common source of confusion -- see the confusion matrix in reports/eval.md. Class balance in training data skews neutral.

License

Adapter weights: Llama 3.2 Community License (this model is a derivative of Llama 3.2 3B Instruct -- see the license and acceptable use policy). "Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved." Built with Llama.