mistral-7b-quantized-lora-finetuned

Model Card for gherke/mistral-7b-quantized-lora-finetuned

<!-- Provide a quick summary of what the model is/does. -->

This is a financial sentiment analysis model, fine-tuned for fine-tuned for sentiment analysis to return a sentiment score between -1 (very negative) and 1 (very positive). It was specifically trained to analyze financial news and assess its impact on financial market trends.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is a quantized version of the Mistral-7B model, fine-tuned using the LoRA (Low-Rank Adaptation) technique for sentiment analysis tasks. The model was trained on the takala/financial_phrasebank dataset to detect sentiment related to economic or market-relevant information. The output is a single sentiment score, with values between -1 and 1, representing very negative to very positive sentiment respectively.

• Developed by: Gabriella Herke
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: Gabriella Herke
• Model type: Causal Language Model fine-tuned for Sentiment Analysis
• Language(s) (NLP): English
• License: [More Information Needed]
• Finetuned from model [optional]: thesven/Mistral-7B-Instruct-v0.3-GPTQ

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: Link to the Hugging Face Model Repository
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

The model can be used for analyzing financial news and producing a sentiment score that indicates the potential impact on financial market trends.

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

The model can be incorporated into larger financial analysis pipelines or trading bots to assess market sentiment.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model should not be used for general sentiment analysis outside of the financial context, as it was specifically trained on financial news.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model is limited by the nature of its training dataset (takala/financial_phrasebank), which may not be representative of all financial scenarios or market conditions. It may produce biased results if applied to other sectors.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be aware of the risks, biases, and limitations of the model. Careful consideration is needed when applying the sentiment scores in automated trading decisions, as biases in the data can lead to incorrect assessments.

How to Get Started with the Model

Use the code below to get started with the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gherke/mistral-7b-quantized-lora-finetuned"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was fine-tuned on the takala/financial_phrasebank dataset, which contains financial news phrases labeled for sentiment (positive, negative, neutral).

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

The financial news phrases were tokenized and preprocessed using the Hugging Face tokenizer, with truncation applied for long texts.

Training Hyperparameters

• Training regime: 4-bit quantized training with LoRA adaptation
• Learning Rate: 2e-4
• Batch Size: 8
• Number of Epochs: 20

Speeds, Sizes, Times [optional]

Training was performed using an 8-bit paged AdamW optimizer, with gradient accumulation steps set to 4.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on the test split of the takala/financial_phrasebank dataset.

Factors

The evaluation was performed based on the model's ability to accurately predict sentiment labels in financial contexts.

Metrics

Mean Squared Error (MSE) and correlation with human-labeled sentiment scores were used to evaluate model performance.

Results

The model achieved reasonable accuracy in predicting sentiment scores within the financial domain, performing well on positive and negative examples but showing some difficulty in identifying neutral cases.

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA A100 GPU
• Hours used: 15 hours
• Cloud Provider: AWS
• Compute Region: Europe (London)
• Carbon Emitted: Approximately 25 kg CO2eq

Technical Specifications [optional]

Model Architecture and Objective

The model is based on the Mistral-7B architecture, with LoRA applied for efficient fine-tuning in the sentiment analysis task.

Compute Infrastructure

Hardware

The model was trained on a single NVIDIA A100 GPU with 40 GB VRAM.

Software

• Transformers Library: Hugging Face Transformers v4.31.0
• PEFT Library: v0.12.0

More Information [optional]

For further questions or inquiries about this model, please reach out to Gabriella Herke.

Model Card Contact

For more information, contact Gabriella Herke at [contact information].

Framework versions

• PEFT 0.12.0