financial-advice-llm-Llama-3.1-8B-Instruct

Financial-Advice-LLM: A fine-tuned Llama 3.1 Model for Structured Budgeting and Financial Advice

Introduction

This project fine-tunes Llama-3.1-8B-Instruct to generate structured, personalized financial advice based on a user's financial profile. While general-purpose LLMs often produce inconsistent or unstructured financial recommendations, this model addresses that gap by training on a large synthetic dataset designed to enforce format, clarity, and completeness. The goal is to help users better interpret their financial situations and receive actionable, template-aligned guidance. After fine-tuning, the model demonstrated improved structure and completeness in budgeting tasks while retaining competitive general reasoning performance.

Data

Because no public dataset exists for personalized budgeting guidance, I created a synthetic dataset of over 300 examples using few-shot prompting. Each example includes a structured instruction (income, debt, family size, goals, etc) paired with a detailed budgeting response covering:

1. Budget Overview
2. Savings Recommendations
3. Debt Strategy
4. Additional Tips

The dataset follows a consistent Instruction / Response format and a 90/10 train and validation split was used to prepare the data.

For evaluation, I used:

• GSM8K-COT for math and numerical reasoning
• MMLU High School Math for general reasoning
• TruthfulQA (MC1) for factual robustness
• A custon LLM-as-a-judge dataset to evaluate structure, completeness, and accuracy of budgeting outputs

These benchmarks collectively measure both general reasoning and task-specific financial performance.

Methodology

Based on experimentation through homework assignments, I selected full fine-tuning because lightweight approaches were insufficient for enforcing consistent multi-section templates. Full fine-tuning allows deeper alignment to structural patterns found in the synthetic dataset.

Three hyperparameter configurations were tested:

The final model was selected from Configuration 2, which showed the lowest validation loss and most stable learning.

Evaluation

Benchmark Results

Benchmark Task Descriptions

I selected these benchmarks to test both general purpose and domain-specific reasoning:

• GSM8k: Makes surer the model still performs well on multi-step numerical reasoning tasks, relevant for budgeting math
• MMLU High School Math: Checks general reasoning retention after specializing
• TruthfulQA (MC1): Assesses factual robustness and resistance to misleading questions
• LLM-as-a-Judge: Measures template adherence, accuracy, and completeness

Summary of Performance

As expected, the model shows a slight decrease in general reasoning benchmarks because of specialization for budgeting style responses. To add on, our third benchmark, TruthfulQA improved modestly, showing that the fine-tuning did not harm general factual robustness. However, the LLM-as-a-judge scores improved in completeness and accuracy and stayed consistent in structure, meaning the fine-tuned model is better aligned for its intended task.

Usage and Intended Uses

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

model_name = "db5kb/financial-advice-llm-Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

prompt = (
    "Instruction: I earn $85k, have $10k in student loans, and want to save for a home.\n\n"
    "Response:"
)

pipe(prompt, max_new_tokens=350)

Intended Use Cases

• Personalized budgeting and financial advice summaries
• Beginner-friendly explanations of financial principles
• Consistent template-based financial guidance

Not meant for investement decisions, tax planning, or high-stakes financial advice

Prompt Format

The model expects inputs formatted as:

Instruction: <user financial profile>

Response:

Example:

Instruction: I'm a single professional earning $75k with $20k in student loans and want to save for a home.

Response:

Expected Output Format

The model outputs a structured, multi-section budgeting summary:

1. Budget Overview:
<text>

2. Savings Recommendations:
<text>

3. Debt Strategy:
<text>

4. Additional Tips:
<text>

Limitations

• The dataset is fully synthetic and may miss some real-world scenarios and variability
• The model may oversimplify complex financial situations
• There are slight performance decreases on general reasoning benchmarks
• Evaluation through an LLM judge may introduce bias
• It should not be used for financial decision making