Llama-3-8B-Therapy-PEFT

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

A LORA fine tuned version of Llama 3 8B instruct that is meant to serve you as a outlet to your negative thoughts

• Developed by: John4Blues (Alt account for 9Skies)
• Finetuned from model: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
• Demo: https://huggingface.co/spaces/John4Blues/Therapy_Llama_3_8B

Risks and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• By no means is this suppose to replace a therapy counselor, please seek professional help if you believe you need it.

• The responses from the model may not be factually accurate, please double check with other sources when you believe you need to.

How to Get Started with the Model

Use the code below to get started with the model.

The LORA/PEFT has already been merged with the uploaded model.

from transformers import AutoModelForCausalLM
model_id = "John4Blues/Llama-3-8B-Therapy"
model = AutoModelForCausalLM.from_pretrained(model_id)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

1. Amod/mental_health_counseling_conversations
2. mpingale/mental-health-chat-dataset (processed)

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

google colab notebook

Training Hyperparameters

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

• batch_size: 2
• gradient_accumulation_steps: 2
• epochs: 3
• learning_rate: 2e-4
• warmup_ratio: 0.03
• dtype: fp16