finance-llama-3.1-8b

finance-llama-3.1-8b

Model Description

This is a fine-tuned version of Meta's Llama-3.1-8B model, specialized for financial question-answering and advice. The model has been trained on financial instruction data to provide better responses to finance-related queries.

Training Details

• Base Model: meta-llama/Llama-3.1-8B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Framework: Unsloth
• Dataset: Josephgflowers/Finance-Instruct-500k (subset of 10,000 samples)
• Training Time: ~2 hours
• Final Training Loss: 0.9580

Training Configuration

• Learning Rate: 2e-4
• Batch Size: 2 (per device)
• Gradient Accumulation Steps: 4
• Max Steps: 100
• LoRA Rank: 16
• LoRA Alpha: 16

Intended Use

This model is designed for:

• Financial question answering
• Investment advice and guidance
• Tax-related queries
• Retirement planning discussions
• General financial education

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("gandhiraketla277/finance-llama-3.1-8b")
tokenizer = AutoTokenizer.from_pretrained("gandhiraketla277/finance-llama-3.1-8b")

# Example usage
prompt = "User: What is the difference between a tax credit and a tax deduction?\n\nAssistant:"
inputs = tokenizer.encode(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=200,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response[len(prompt):])

Example Queries

The model can handle various financial topics:

• Tax planning and deductions
• Investment strategies and portfolio diversification
• Retirement planning (401k, IRA, etc.)
• Debt management
• Financial terminology explanations

Limitations

• This model provides general financial information and should not be considered as professional financial advice
• Always consult with qualified financial advisors for personalized guidance
• The model's knowledge is based on training data and may not reflect the most current financial regulations
• Not suitable for high-stakes financial decisions without professional verification

Training Dataset

The model was fine-tuned on a subset of the Finance-Instruct-500k dataset, which contains instruction-following examples focused on financial topics.

Evaluation

The model shows improved performance on financial question-answering tasks compared to the base Llama-3.1-8B model, with more domain-specific knowledge and better structured responses.

Ethical Considerations

• This model should complement, not replace, professional financial advice
• Users should verify information and consult qualified professionals for important financial decisions
• The model may reflect biases present in the training data

Technical Details

• Architecture: Llama-3.1-8B with LoRA adapters
• Precision: 4-bit quantization during training
• Framework: Unsloth for efficient fine-tuning
• Hardware: Trained on GPU with CUDA support

Citation

If you use this model, please cite:

@misc{finance_llama_3.1_8b,
  title={finance-llama-3.1-8b: A Finance-Specialized Llama-3.1-8B Model},
  author={gandhiraketla277},
  year={2025},
  howpublished={\url{https://huggingface.co/gandhiraketla277/finance-llama-3.1-8b}}
}

Acknowledgments

• Meta AI for the base Llama-3.1-8B model
• Unsloth team for efficient fine-tuning framework
• Finance-Instruct-500k dataset creators