fingpt-forecaster-llama3-lora

FinGPT-Forecaster: LLaMA-3-8B-Instruct with LoRA Fine-Tuning

The model takes a prompt with stock information — company metadata, recent news headlines and summaries, basic financial metrics, and 2-week price history — and predicts the company's stock price movement for the next week.

Output follows a fixed schema.

Model Sources

• Repository: https://github.com/AI4Finance-Foundation/FinGPT
• Base Model: meta-llama/Meta-Llama-3-8B-Instruct
• Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

---

Training Details

Training Data

Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405

Covers all 30 DOW Jones Index constituents over May 2023 – May 2024. Each sample is a weekly snapshot including:

• Company profile — name, sector, IPO date, market cap, exchange, ticker
• News — headlines and summaries from the past 1–4 weeks (via Finnhub)
• Basic financials — ~35 ratios (EPS, P/E, gross margin, ROE, debt ratios, etc.)
• Price history — weekly open/close prices for the lookback window
• Answer — structured output with positive developments, potential concerns, a % price movement prediction, and analysis (GPT-4 generated labels)

Data is sourced from yfinance and Finnhub. Split: 80% train / 20% test.

---

Training Procedure

Supervised fine-tuning (SFT) of LLaMA-3-8B-Instruct using LoRA. Dataset prompts are originally in LLaMA-2 chat format ([INST]<<SYS>>...[/INST]) and are automatically converted to the LLaMA-3 chat template via tokenizer.apply_chat_template before tokenization. Prompt tokens are masked from the loss; only answer tokens are supervised.

Preprocessing

• Tokenizer: meta-llama/Meta-Llama-3-8B-Instruct; pad_token = eos_token, padding_side = right
• Prompt format: LLaMA-3 chat template (system + user messages), with add_generation_prompt=True
• Max sequence length: 8192 tokens
• Samples exceeding max length are filtered out before training
• LoRA parameters explicitly cast to fp32 for gradient scaler compatibility

Training Hyperparameters

LoRA Configuration:

Speeds, Sizes, Times

• Hardware: Multi-GPU SLURM cluster (4 GPUs per node), Apptainer container
• Checkpoint cadence: Every 10% of training steps

---

Evaluation

Metrics

Evaluation is run at each checkpoint via GenerationEvalCallback on 50 randomly sampled test examples (greedy decoding, max 512 new tokens, stopping at <|eot_id|> or EOS).

Parse rate (% of outputs matching the expected structured format) is also logged.

Results

---

Technical Specifications

Model Architecture and Objective

• Base: LLaMA-3-8B-Instruct (decoder-only causal LM, 8B parameters)
• Adaptation: LoRA on all attention projection layers and MLP gate/up/down projections
• Objective: Next-token prediction (cross-entropy) over the structured answer only; prompt is masked
• Key difference vs LLaMA-2 version: Uses the native LLaMA-3 chat template and <|eot_id|> as an additional stop token; gradient checkpointing uses use_reentrant=False

Compute Infrastructure

Multi-GPU SLURM cluster with Apptainer containers. Training coordinated via torchrun with NCCL backend and DeepSpeed ZeRO-2. Weights & Biases for experiment tracking.

Hardware

• NVIDIA A100 GPUs (4 per node)

Software

---

Citation

BibTeX:

@misc{fingpt2023,
  title={FinGPT: Open-Source Financial Large Language Models},
  author={Yang, Hongyang and Liu, Xiao-Yang and Wang, Christina Dan},
  journal={FinLLM Symposium at IJCAI 2023},
  year={2023}
}