Llama-3.1-Omni-FinAI-70B

Llama-3.1-Omni-FinAI-70B Model Card

Model Overview (Built with Llama)

Llama-3.1-Omni-FinAI-70B is a pre-trained large language model optimized for finance-specific fine-tuning applications. Based on the LLaMA 3.1 70B architecture, this model was pre-trained on 143 billion tokens of high-quality financial texts. Llama-3.1-Omni-FinAI-70B provides a foundation for further fine-tuning in specialized financial analysis tasks.

Model Details

• Base Model: Llama-3.1-70B-Instruct
• Training Data:
  • SEC 10-K, 10-Q, and 8-K filings
  • Reuters News data (RCV1, TRC2)
  • Finance-specific papers from Arxiv
  • Financial discussions from Reddit
  • Wikipedia
• Primary Use Case: Pre-training for finance-specific fine-tuning, allowing users to leverage Llama-3.1-Omni-FinAI-70B's foundational financial language understanding.

Use Cases

Llama-3.1-Omni-FinAI-70B is designed as a base model for finance-specific fine-tuning tasks, supporting applications such as:

• Sentiment Analysis
• Stock Movement Prediction
• QA Instruction
• Summarization
• Predictive Financial Analysis

Training Process

Llama-3.1-Omni-FinAI-70B was trained using the NVIDIA NeMo framework on 64 H100 GPUs, utilizing a diverse dataset that ensures robust performance for fine-tuning in finance-related applications.

Limitations

This model is pre-trained for finance-specific fine-tuning tasks and may require additional fine-tuning for specialized applications. Due to its large size, substantial computational resources are recommended for deployment.

License

This model is licensed under the Llama 3.1 Community License.

Citation

If you use the Llama-3.1-Omni-FinAI-70B model, please cite as follows:

Chiu, I-Chan and Hung, Mao-Wei and Chen, Zih-Ching and Chiu, Jun-wei and Lin, Yang-Hsien and Lee, Cheng-Kuang and Huang, Eddie TC and See, Simon, Omni-FinAI: Unlocking Financial Disclosure Insights (October 30, 2024). Available at SSRN: https://ssrn.com/abstract=5004298